Managing Complexity
Presented by Larry Garfield (@Crell)
@Crell
- Senior Architect, Palantir.net
- Drupal 8 Web Services Lead
- Drupal Representative, PHP-FIG
- Advisor, Drupal Association
- Loveable pedant
Disclaimer: Many external references
See session page for full list
What is complexity?
Simple
- From "Simplex"
- One braid
- Opposite: Complex
Easy
- From "aise"
- Lie near
- Opposite: Hard
Simple is a synonym of Decoupled
Easy is a synonym of "familiar"
Complect: To braid together; make complex
Decomplect: To unbraid, separate
Compose: To place together (not mix)
Simplicity is a prerequisite for reliability.
—Edsger W. Dijkstra
Simplicity is the ultimate sophistication.
—Leonardo da Vinci
Programming, when stripped of all its circumstantial irrelevancies, boils down to no more and no less than very effective thinking so as to avoid unmastered complexity, to very vigorous separation of your many different concerns.
—Edsger W. Dijkstra
We as humans are very limited in our ability to deal with complexity but unlimited in our ability to create it.
—Sam Boyer
When you build on something you inherit the complexity of the lower layers.
—Sam Boyer
The only way to manage complexity
is to eliminate it.
What is scale?
Norris Numbers
- < 2K LOC: Any code will do
- 20k LOC: Need classes, packages
- 200K LOC: Need vigorous focus on simplicity
- 2M LOC: ??? (functional principles?)
Complexity/LOC
Simple code has linear complexity
Complected code has geometric complexity
Democracy
- Consensus: A dozen or two
- Direct: A few hundred
- Republic
Dictatorship
- Chieftan: No nobles
- Kingdom: Nobles with fiefdom
- Empire: Local governors
The approach at one scale is entirely different than the approach at a higher scale.
Structure
The Tyranny of Structurelessness
—Jo Freeman, 1970
We cannot decide whether to have a structured or structureless group, only whether or not to have a formally structured one.
Structure is inevitable
Formal
- Documented
- Clear change points
- May be simple or complex
- Leads to process
Informal
- Implicit
- Changes without notice
- May be simple or complex
- Leads to cliques/elitism
Elitism
Elites are not conspiracies... Elites are nothing more, and nothing less, than groups of friends who also happen to participate in the same... activities... It is the coincidence of these two phenomena which creates elites..
—Jo Freeman, The Tyrrany of Structurelessness
Elites are informal structure
Informal structure fills in a vacuum in formal structure.
Informal structure creates informal qualifications.
Informal structure complects with irrelevancies.
Successful informality
- Task-oriented
- Small and homogeneous
- High communication
- Low specialization
Problems
- Who's in charge?
- Who's responsible?
- Who can veto/approve?
- What backchannels are there?
- How do I get "in"?
- How do I retire?
- When has leadership changed?
If the movement continues deliberately to not select who shall exercise power, it does not thereby abolish power. All it does is abdicate the right to demand that those who do exercise power and influence be responsible for it. If the movement continues to keep power as diffuse as possible because it knows it cannot demand responsibility from those who have it... it insures that the movement is as ineffective as possible.
—Jo Freeman, The Tyrrany of Structurelessness
There is no such thing as structureless.
Informal structure is extremely complex & inefficient.
—Ashe Dryden, 2013
Meritocracy is the belief that those with merit float to the top - that they should be given more opportunities and be paid higher.
—Ashe Dryden
Problem
"Merit" doesn't exist in a vacuum
- Caregiver of children/family
- Medical conditions
- Long commute
- Employment contracts
Do-ocracy
"One hour, one vote"
Rule by those with nothing better
to do with their time
How does this apply to us?
Who is Drupal's Tech Lead?
What does an Initiative Lead do?
Initiative Leads have the following responsibilities:
- Coordinate work on the initiative.
- Communicate the plans, progress, and needs of the initiative to the community and the branch maintainers.
What does a component maintainer do?
Responsibilities of component maintainers include:
- Issue queue farming
- Mentoring
- Point of contact
- Reviewing patches
- Answer questions
- Write actionable issues
What does a branch maintainer do?
Branch maintainers oversee the development of a major version of Drupal, by making the ultimate decisions on which patches are applied to the software, and by working with the community to set goals, priorities, and guidelines.
Who is Drupal's Tech Lead?
@da_wehner Drupal Technical Lead: Person with most patches this week.
— Paul Mitchum (@PaulMitchum) September 3, 2014Who was Drupal's Tech Lead?
Who do I need to run this issue by?
—New contributor
Answer: Who knows?
If the movement continues deliberately to not select who shall exercise power, it does not thereby abolish power. All it does is abdicate the right to demand that those who do exercise power and influence be responsible for it. If the movement continues to keep power as diffuse as possible because it knows it cannot demand responsibility from those who have it... it insures that the movement is as ineffective as possible.
—Jo Freeman, The Tyrrany of Structurelessness
Component maintainers are ignored daily
Why is X not done?
If you're a developer, and you aren't fixing #Drupal 8 critical bugs, care to share why? Curious what we can do to help momentum.
— Angie (@webchick) September 5, 2014@webchick I have no desire to spend time attempting to disagree with someone who likes sleep less than me.
— Matt Kleve (@vordude) September 5, 2014@webchick my mental health is worth more to me than core's release date
— Greg Dunlap (@heyrocker) September 5, 2014Do-ocracy
Rule by those with nothing better
to do with their time
"Easy" also means "easily replaceable", unskilled labor.
The result
- No one has authority
- Begging for the bikeshed
- Steamrollers win by doing it the easy way
- Easy is not simple
- Informality at scale is horribly complex
Drupal the code base inherits all of
the complexity of Drupal the community
organizations which design systems ... are constrained to produce designs which are copies of the communication structures of these organizations
—Conway's Law, 1968 (Melvin Conway)
The only way to manage complexity is to eliminate it.
Clear
Decoupled
Structure
How?
More than framework vs. app
Joomla did that. It was bad...
Largest OSS projects
- Linux
- KDE
- Apache
- Eclipse
- Drupal
- Gnome
- Perl+CPAN
- Mozilla+Addons
- GNU
10x larger than others (2010)
Commonalities
- Community projects, not corp
- Foundation governance
- Highly modular/collections
Linux
- Linus doesn't merge patches
- Component maintainers merge patches
- Linus merges changesets
- Most components are decomplected
Drupal needs to be a composition of components,
not a complection of components
The code and the community.
Respect subsystem maintainer authoratah!
But how do we do patches across modules/systems?
That we need to is the problem...
- "Simplifier" role
- Keeper of simplicity/quality in focused area
- Ignore release dates, global strategy
- Not interested in "features"
- Partner with "Easy" people
Brutal focus on simplicity
At all levels
Even at the cost of easy or features