Methodologies for Skilled Innovation

Here to summarize my experience in how to do innovation in the daily engineering work. The focus area is distributed storage systems.

Common Senses

First, why engineering work needs innovation?

• Incremental innovation. Our daily work is to build better systems. The work is essentially composed of incremental innovation. Some proposed new features enroll into next level paradigm shift, or even disruptive innovation. Compared to infrastructure systems, App development receives more drive from changing business needs.

• If the product aims as a leader in a market unit, innovation is a must, to explore what’s next, and more to construct the next market rules. If the product is a follower, the task of innovation is more shifted to learn from competitors and eco-systems, to keep up with “what others are playing”, and to regularly attend in info sources.

• Innovation is a fundamental thought training (along with solving problems, broad reading & writing). Let it train our insight depth, sensitivity to subtle opportunities, grasp of the system overview, and knowledge to industry.

The common sense innovation method is brainstorming and idea addition/subtraction/multiplication/division. Usually most ideas will be filtered out, but the surviving one in ten can be shiny.

• Besides free style, brainstorm ideas but with structured categories and heuristics, mine ideas with MECE, and combination iterations.

• Besides, sleep and forget. Let those ideas mutate and reform in subconsciousness. When come back and look again, new discoveries emerge.

Methodologies for Analyzing

The fundamental method of innovation is to stand on the mountain top and look one step further. Step two is easy.

• Step one. Be the expert. Master the technology stack. We should be able to answer anything when asked. Know where the mountain top is, industry and academic, and the underwater momentum. Besides, understand the product in depth, the industry landscape, and our position. This is non-trivial, which should take years. [1][2].

• Step two. think what should be the next for the industry and the product. Review our product and think what are the bottlenecks in the near future. What can be the threats. And what are the strength to enhance. These gaps are where to find the innovation. Apply visions and strategic thinking.

Gaps analysis are convenient source of innovation.

• Observe what people are complaining about, what are the problems open not solved, what are we today and we want to be. These map to gaps. Analyze into the gaps, explore solving problems; they usually bring innovation.

• SWOT analysis helps understand gaps and threats. We can analyze comparing with market competitors, eco-systems, and opensource communities, to understand where we are and what are the gaps.

• Min-Max analysis. A gap maps to a problem. We may not solve every problem, but we should always be able to plot it on a continuum axis, and scope it with what we are the min-max. We can plot the theory limits vs where we are to find the gaps. We can plot the reaching scope by today’s ability vs in following future, to identify what we should reach but didn’t, what we cannot, and what to prepare in future.

  • Breakdown. Breakdown the topic in to fundamental elements, and then recompose them to analyze how far they are capable to reach in theory. This is similar to First Principle Thinking.

• Grid search. Continued from Min-Max, we can unfold the properties we care into multi-dimensions of structured space. Use an exhaustive method to divide and categorize the space, and conduct the search. This also helps us find and look into possibilities. In another word, we should have the map, even we cannot reach every location, and we should plot grid axes on the map.

• Review by doing. When review a doc, design, proposal, or code, close your eyes and think how you yourself would do it first. Compare your solution with the one being reviewed, your will always find points to innovate. Similarly, use the technique to review how we do things today, how the current system is built, to find innovation.

Essence analysis is another way of innovation.

• Breakdown idea, strip away the unnecessary part. What are left are the essence. Essence are transformed mapping of the original problem. Essence reflect the core info and properties that are passing around while holding the invariant and equivalent.

• Recombination of the essence will generate new ideas. Add, subtract, multiple, division happen here to combine them.

Thinking in high level, thinking in abstraction.

• High level and abstraction. A easy pitfall when newly joining a project is to see a series of tasks, issues, bugs to work on. You find small pieces of new things to work on, but no big and impactful components. To come out of this situation, you need to think in high level, think in abstraction, and think with overall system understanding. Don’t solve one individual problem, but instead try solving the generalized a group of problems. Use 5 whys to further dig what’s the underlying generic problem.

Sources for Innovation

Problems & missing parts in engineering daily work are source of innovation.

• Online service and operation issues. Apply systematical thinking and in depth. An issue to happen usually reflex missing functionalities, gaps to fill, and opportunities for paradigm shift. “5 whys” is useful method from Critical Thinking, e.g. why the failure is not detected before impact, why auto mitigation not working, why it is captured now rather than at an earlier stage, etc.

• Don’t repeat yourself. If we are working with repeated issues, with repeated manual operations, with repeated coding, that usually identifies tooling innovations, paradigm shift how we work, or opportunities to change how we share and reuse.

• Frequently we meet that, want to do A, found A needs B1, B2, B3 first. And then Bi needs again Ci1, Ci2, Ci3 first. This is work blocking but chances for innovation. Think in higher level and abstract general, this usually brings new components to address general needs. Think about the history of NASA and Internet from military.

• The preset for innovation is to measure it first. Not only abundant metrics, logging, analytics sources help troubleshooting, and help identify the gaps we have. A better measure in finer grain usually reveal what we don’t know before, and breaks the wrong hidden assumptions. These then become new places to improve.

There are wiser views to manage product and scope to help innovation

• Instead of only focusing on own product, we should reach into the customers about how they use and think. This helps identify the real problems to innovate from.

• Instead of only focusing on own product, we should extend the scope to understand and learn from surrounding products and teams. There can be things we missed; there can things we can leverage.

• Keep updated with new feature or services released. The surrounding team may recently build new technologies, added new features, or running new services. We can think how to leverage them to improve own product. We can think how to apply them to new problems and new scopes. This are often sources of innovation, and is well welcomed by other teams.

• Find better alternatives and improvements. When we face the existing features surrounding us, we can routinely think are there better ways, or can they be improved? Such thinking also generates innovation. Also, review our product, where we are internally & externally, in a holistic view, are thought training and knowledge accumulate that make innovation more fruitful.

Team working is important for innovation

• It’s non-trivial work to keep updated and follow up all technology directions. Build a team, where each member covers a different direction, and do regular share sessions like seminars. It will more efficient.

• Teamwork helps inspire new ideas, e.g. brainstorm is not running on individual. Ideas passing back-forth, listened and reflected, can usually generate much more.

• Idea polishing needs teamwork. A new idea needs to be verified. It needs to be extended. It needs to be patched for weakness. Sometime it needs to be turned into another way, or brought into a new level. This usually happen across a team, which idea passing back-forth, interactively, challenge and get challenged. Everyone has the unique experience to help idea grow, to avoid it stuck in a dead end.

• Besides working in office, innovation also needs info feed from the external world. We can participant regularly in meetups, summits, etc, to meet guys from different places and how they solve related problems. This is usually key to know our industry position, catch up and ensure we are not hitting in a wrong direction.

• Help other people, and answer questions. Besides providing knowledge, answer questions provide chance to review our knowledge and product again, and understand how people see it from another perspective. This can usually find new gaps. And it broadens our scope and helps other people grow, eventually a positive feed for innovation.

Visions & Strategy

Visions & strategic thinking that help innovation

• Innovation can be inspired from vision thinking, e.g., what are we doing for the next 1~3 years, and 5~10 years, what will our product be like, what are the technology directions and our position, internally or externally.

• In another perspective, in the next 3~5 years, how we will be doing daily work and organize the team. What are the key driving factors behind, the bottleneck, and the growth space.

• What is the lifecycle and position of our product, our technology domain, and our market. Innovation is generally having more opportunities in the emerging new domains, more challenging areas, and generally less easier in the mature and fading markets.

• Read in a systematically way and compare. It’s comparing rather than linear reading that reveals intrinsics. Innovation comes from the compare.

• Following industry trends, watch big market vendor/leader moves, study venture capital news. This helps find emerging but not proven technologies on the climbing curve.

• Challenge how we do things. If the answer is we were doing it in the way we did it before, this is a bad smell. Use analytics skill to analyze the rationales behind and the gaps to a changing today, you will always find a source of innovation. This point is also connected with Review by doing.

• Watch for new technologies. They can be applied to your daily work or current system and brings innovation. A typical example is LLM and GPT.

• Disruptive innovation. Rather than cumulative improvements following the existing path, true innovation is disruptive. It comes by jumping out of the box, or say a paradigm shift. It’s disruptive because it breaks how system works today, the day 1 product is usually weaker than today, but with a faster growth rate, a higher instinctive efficiency, and eventually the higher ceiling limit. Market growth starts from replacing exiting technology, and then harvest the new space created. New jobs are created in the replacing process by rebuilding today’s system. Industry level is pushed forward with the new space. Instead, cumulative improvements suffers from diminishing growth potential, eventually leads to involution.

Driving Factors in Storage Area

In the end, here lists the storage driving factors summarized from scratch. It can help plot the innovation landscape.

Hardware

• New type of hardware, e.g. 3D-XPoint, SSD, SMR, ZNS, NVMoF.
• Growing capacity of hardware, e.g. TB memory, 1 machine 100+ HDD, manycore, RDMA.
• Changing composition of hardware, e.g. colocating compute/storage vs disaggregated design in Snowflake, faster networking / faster storage device vs catching up with CPU.
• Smart hardware, e.g. FPGA, Computational Storage, AWS Nitro, Azure Catapult, ASIC.
• GPU, DPU, IPU, Smart NIC, ARM.
• QLC, NVMoF.
• Share-everything architecture.

Growing scale of data capacity and IOPS

• Distributed scale-out storage, Ceph, now everything.
• Distributed filesystem, CephFS, EMC Isilon, Azure HDFS/DataLake.
• Cloud opening hundreds datacenters worldwide, interconnect, geo-distribution.
• Performance deterministic, SLA

COGS saving

• Erasure coding, performance optimization, Kernel pass-through.
• Data deduplication.
• Data tiering, migrating, caching, etc.
• Underlying data representation, data format, physical layout.

Reliability

• Data backup, Copy data management.
• Practical formal testing. E.g. TLA+, AWS S3 ShardStore.

Related technology

• Container, cloud.
• TLA+, formal verification.
• Consistent hashing, distributed transactions, append-only LSM tree, ART / Mass-tree indexes.

User interface changing, new business models

• SQL, No SQL, New SQL.
• OLAP, OLTP, HTAP.
• In-memory storage, All Flash, NVMe storage.
• Cloud, Hybrid Cloud, Cloud offloading, Hyper-converged, IoT, Edge computing, Blockchain.
• Pushes from new consumer segments, e.g. the active Database community, VM & virtualization, NFS, page & block storage, archiving, data backup, AI & machine learning & deep learning, GPU, Datalake & enterprise data management and analysis, software defined DC/storage, containerization, etc
• Stream processing, transactional streaming, evolving table, RDD, XOR linage.
• Video processing. online realtime podcasting.

Co-design with surrounding systems

• Co-design with database some years, and then decouple design some years.
• Co-design the custom hardware some years, and then decouple with whitebox commodity hardware some years.
• Open-Channel SSD, ZNS GC, SMR GC.

Security

• Zero trust, confidential computing.
• Ransomware, Immutable Storage.

Data management

• Operational model changing, Orchestration.
• On-premises and hybrid cloud management and COGS improvement, metrics and analytics COGS saving.
• Data migration, shipping, Edge, caching, Tiering.

Other Tips

Tips for how to work with innovation.

• Time to stay alone uninterrupted is extremely important. It triggers active mind thoughts. Innovation frequently happens when being alone, relaxed with doing simple non-mind-intensive labor, e.g. taking shower, casual reading, walking. Especially it’s after long hours of intensive thinking, ingesting new info, or discussing with people for rounds. Sleep well, sleeping is another kind alone and idle.

• Don’t get too busy. Free relaxed time is important. Sleep more than enough to let your mind reorganize itself. To have a breakthrough idea, it doesn’t take time, it takes space.

• Separate your fragmented time and big block time. Phone browsing gets updates but it is time consuming. Only phone with fragmented time. Big block time should do big block things.

• Changing the living environment helps. Stay at a new place for a few days to get rid of stereotypes, fixed thinking mindsets, and distracting daily errands.

• Always discuss with more people. Involving 2~3 people to discuss and bouncing can drastically improve the idea quality and bring up new opportunities.

Useful tips from more sources

• Learn from history, understand upstream and downstream, and stand on shoulders of giants. Summarize and analyze down experiences and issues systematically (归纳和总结). Dig to bottom rather than solve case by case (治本而不是治标). 局部、全局，以及发散的眼光，前瞻未来的问题。 发现问题、眼光，定义问题。 Architecture Thinkings by 阿里道延

• Practice writing and presenting to others. Clarify and make sure you only present what you understand, and make sure the audience can understand. Writing means clarifying your mind; Always try to write something – some of the best ideas will come to you by writing. Instructions for PhD Students - Dimitris Papadias, HKUST