Conway’s Law and the Shape of Organizations
If you aren’t familiar with Conway’s Law, it’s an adage introduced in 1968 by author and computer programmer Melvin Conway.
Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization’s communication structure. – Melvin E. Conway
There have been numerous variations of this law over the years, but they all basically boil down to the same ideas. If you have a large, monolithic organization with a top-down, command-and-control hierarchy, you’re information systems will reflect this type of structure. If you have an organization built of loosely-coupled, agile teams, your information system will reflect this type of structure.
I won’t go into the pros and cons of these different organization structures, other than to say that they each have their purpose. Things get tricky when one attempts to play in the other’s playground. Nimble innovation teams do not do well when faced with difficult domains that require many policies and procedures to meet regulation and compliance. Large monolithic organizations tend to break agile practitioners against rocks, kill their spirits, and then claim that agile practices don’t work. Of course not, innovation can’t happen in a commodity environment. It’s nearly impossible. And attempting to innovate with highly regulated fields is asking for headaches.
The general understanding of most system architects is that if you want a different type of information system, you need a different type of organization. Break down your monolith organization into smaller agile teams and your software will better reflect this.
But what happens if you can change the information channels? Conway’s law was based on the idea that communication channels are what define the architecture of the organization. What if those communication channels are no longer limited to the senses of individuals and teams within the organization? What if the organization itself, not just the people within, begins to develop its own sense and intelligence? Does Conway’s Law still apply? My initial thoughts are yes, but I want to dive into “why?”
The Challenge to Innovate
What are the challenges that organizations face today when they want to undergo a digital transformation, or agile transformation, or whatever you want to call it? Well, to start, I think these organizations need to ask themselves why they think this transformation is necessary?
Of the organizations I’ve been involved with, the problem is often silos, and the push for change is rarely coming from the executive level, but from lower-level teams who have been tasked with taking on innovation. The request for innovation comes from the top, but there’s rarely any real room made in the organization for this innovation to take place. It’s almost as though some leaders think innovation just pops into existence.
The thinking is that these lower level portions of the organization can just implement agile transformation on their own, without the need for cross functional communications or any changes from leadership. Leadership’s dashboards are set with metrics that fit their view of the organization and any attempt to steer away from that is often met with pushback.
And though many CTO’s job descriptions includes ‘innovation’ as a requirement of their position, if the choice is between keeping the team focused on “lights-on” activities that continue to produce business value or directing budget and resources on a nebulous “innovation,” strategy, well, they’ll keep the lights on. That’s a no-brainer. CTOs are fired because of operational failures, not because they didn’t invent the next Facebook for the manufacturing industry.
What are teams, really?
Siloed departments are often thought of as teams, but that is a misnomer, because these siloed departmental “teams,” are not fully autonomous. For a team to truly function as an innovation center, they need autonomy.
For example, let’s say I’m the marketing director for my organization. I have a team of marketing professionals, but I don’t have access to the technology team—the IT department has better things to do, like create dashboards for the executives. I need to manage websites, social media, email marketing. My team includes designers who work on Macs and they produce great looking image-based assets. A few of my team members even know enough HTML and JavaScript that they can create an impressive landing page. However, they don’t have direct access to the web servers and other services that help provide my marketing platform. Without building out my own shadow IT department, I won’t be able to hit some of the innovation goals on my bosses’ lists. Mobile App? Maybe we can outsource it, but what if we need data from the company’s central database? How will that even work? As the marketing director, how much am I supposed to know and understand about the underlying technology? How much should my team know? Can I bring in consultants to help or will that kill my regular marketing budget?
Siloed departments are not the type of teams who can deliver on innovation. They can deliver, but it better be within some scope of expectation. And scope of expectation is a hard conversation to have with some executives.
So how then do we take my situation, revise my organization, and make my marketing department a center for innovation?
We reverse Conway’s Law.
If my department’s goal is to become an innovation center in marketing, I’ll need a far greater understanding of my organization as a whole. Most importantly, I’ll need an understanding of our customers, how we service our customer’s needs, and what tasks and systems go into servicing those needs. I won’t go into great detail on how that’s done here, but I will direct you to Wardley Mapping. It’s the perfect place to start if you don’t actually know where you are.
Once I understand where my department best services our customers’ needs, and I know what I can outsource, leave to more mature teams internally, and where I need to innovate, I can now build a team around that innovation. That team could include data engineers to help build data capture systems for analytics, mobile app developers to build out a new mobile platform for direct communication with customers, a chatbot to manage answering frequent questions on social media sites. I will need a budget, I’ll need a feedback loop that tells me I’m on the right path. I’ll need direct conversations with customers.
I no longer have a traditional marketing team. I now have an innovation team that takes on marketing challenges. And not all of these challenges are necessarily technology related. Depending on the customer, it’s possible I might need guidance in the form of better human-centered design specialists, design-thinkers, and maybe even an embedded entrepreneur. Once you focus on what you need to deliver to the customer, and you truly understand what you need to deliver to your customer, because their voice and the data is telling you what to deliver, you have a much different team—and that team will need to be dynamic. It will change as your needs change.
Now I have a team that can innovate, because I’m focusing on capturing feedback from my customers with my websites, my mobile apps, and my social network. Valuable feedback comes back into my department in the form of data. My team can analyze data and gain insights that it feeds to other teams within the organization. This in turn helps our services, which helps are customers, and I now have a means of measuring my team’s value.
I haven’t addressed how IoT fits into this. I’ll do that now, but first I’ll need to explain what a sense-enabled company is and how it works.
IoT & AI aren’t new, but they are finally finding their business value
We’re beginning to build an understanding that the old way of forming organizations to do work is not always the best way to build companies who want to remain agile in times of uncertainty or beat their competition with strategic innovation.
Team topology is the true enterprise architecture. Not IT systems. IT systems will never dictate the communication structure of an organization, it will always be the reverse. Humans will go around the rules of the IT technocracy to complete critical business tasks. Ticket systems are ignored if you know a person in the IT department who you can call to fix the problem, because the problem is stopping a mission critical operation and needs to be repaired NOW!
Is this still valid with the addition of extra-sensory perception—and I don’t mean psychics here, but devices that perform as extra eyes, ears, noses, and sometimes senses that are beyond human detection—and even more recently, have taken on the added role of predictors and decision makers?
I won’t discuss deeply what IoT Edge is in this post. Feel free to see some of my other articles here for more context around its capabilities:
IoT in the Monolithic Organization
IoT is not new. Manufacturers were capturing telemetry from industrial equipment (often referred to as operational technology) long before the internet as we understand it was a thing. These telemetry gathering devices tended to be embedded in the equipment, hardwired, and worked on proprietary protocols. Capturing the data was/is managed by factory floor devices called Historians and telemetry was processed in batches. Alerting was all local and tended to be difficult to program. IoT has evolved from the technology of the early days, but it remains similar in function to the older telemetry generating and gathering systems.
And this type of telemetry was fine when your goal was keeping expensive machinery running. That is the goal of many industrial IoT systems today, but that’s not necessarily the goal of many manufacturers. Like most industries, they’ve had to adjust to landscape of manufacturing.
However, when IoT is limited to a siloed department, it’s use is primarily limited within the bounds of that department’s ability to communicate. IoT extends the senses of that department, but the data still isn’t necessarily feeding anyone outside the organization’s communication structure. IoT, when used primarily as a tool within a silo, will likely continue to operate within that silo, and there’s one simple reason for that – IoT is not autonomous. It is merely an extension of the devices and people of that department, so it makes sense that it’s just as siloed as the people.
Let me try to play the Devil’s Advocate here. What about Data Warehouses? Won’t my time-series data end up in a data warehouse? Isn’t that moving beyond the silo?
Okay, so the data might not actually stay on the factory floor in the historian, but is it building a true feedback loop for innovation? No, usually it isn’t. That data is normally used to plan predictive maintenance models, efficiency systems, and possible fill a chart on a dashboard. It’s not so much used to feed the loop, but to burn on the information radiator. So does this data produce value directly for the customer? Not really.
Patterns of IoT and Team Topology
Recent reports show a greater uptick in IoT and AI innovation projects within organizations. However, they still have many of the same challenges of deriving value from these investments. Could it be that their siloed structures are holding them back?
As the number of AI and data experts increase, many industries still want to relegate these individuals to their IT department—the IT department that is far more focused on operational stability than they are on innovation. Which is understandable, most IT departments are given the mandate to keep things operational. Experimentation sounds like a terrible idea when your job depends on security, stability, reliability, you know… known-knowns, absolutely not unknown-unkowns. Keep those machine learning experiments on your laptop using exports from the data warehouse, but don’t you dare try to bring one of those little beasts into production.
So is it any wonder that traditional monolithic organizations who focus on commodity offerings have a difficult time with a data and AI team – a team whose primary focus is on experimentation. Add IoT, a platform often seen as a security nightmare by traditional IT departments, to the mix and you get even more pushback. When you try to tell them that you want to actually take those experimental machine learning models and push them out to IoT device… well, just expect a lot of laughter. Not laughter with you, I’m afraid.
The lack of AI and IoT success doesn’t tend to be with the technology, but with the team structures inside an organization. Again, Conway’s Law comes into effect here. Your information systems will look like your organization. If your organization’s IT department is built like a battleship, don’t expect it to behave like a fleet of speedboats.
So what if an organization wants to reverse Conway’s Law and build teams for Data, AI, and IoT?
I believe that’s the real solution. If an organization wants to implement IoT and AI, they must first recognize a real value for it in their customer value chain. They must then understand how this fits – can that value be supplied by a outsourced product or team? Is a minimal solution ideal for them, like an IoT platform? Or do they need to build a custom implementation with ML on the edge?
I’ve touched on members needed for an AI development team, but I haven’t touched on the concept of Team Topology for innovation teams. I think I’ll do that for a future post.
For now I think it’s safe to say that IoT and AI do follow along with Conway’s Law. However, to get the greatest use out of AI and IoT, monolithic organizations with siloed departments probably need to rethink their structure and team topology.
In my future article I’ll work out what an IoT Edge development team looks like, how it would work as an agency-based team within an organization, and what its innovation stack looks like.
Until then, thanks for reading!