06 November 2021

Here to summarize my experience on how to do innovation in the daily engineering work (in distributed storage system area).

First, why engineering work needs innovation?

  • Incremental innovation. Our daily work is to build better systems. The work is essentially composed of incremental innovation. Some proposed new features enroll into next level paradigm shift, or even disruptive innovation. Compared to infrastructure systems, App development receives more drive from changing business needs.

  • If the product aims as a leader in a market unit, innovation is a must, to explore what’s next, and more to construct the next market rules. If the product is a follower, the task of innovation is more shifted to learn from competitors and eco-systems, to keep up with “what others are playing”, and to regularly attend in info sources.

  • Innovation is a fundamental thought training (along with solving problems, broad reading & writing). Let it train our insight depth, sensitivity to subtle opportunities, grasp of the system overview, and knowledge to industry.

The common sense innovation method is brainstorming and idea addition/subtraction/multiplication/division. Usually most ideas will be filtered out, but the surviving one in ten can be shiny.

  • Besides free style, brainstorm ideas but with structured categories and heuristics, mine ideas with MECE, and comination iterations.

  • Besides, sleep and forget. Let those ideas mutate and reform in subconsciousness. When come back and look again, new discoveries emerge.

The fundamental method of innovation is to stand on the mountain top and look one step further. Step two is easy.

  • Step one. Be the expert. Master the technology stack. We should be able to answer anything when asked. Know where the mountain top is, industry and academic, and the underwater momentum. Besides, understand the product in depth, the industry landscape, and our position. This is non-trivial, which should take years. [1][2].

  • Step two. think what should be the next for the industry and the product. Review our product and think what are the bottlenecks in the near future. What can be the threats. And what are the strength to enhance. These gaps are where to find the innovation. Apply visions and strategic thinking.

Gaps analysis are convenient source of innovation.

  • Observe what people are complaining about, what are the problems open not solved, what are we today and we want to be. These map to gaps. Analyze into the gaps, explore solving problems, usually bring innovation

  • SWOT analysis helps understand gaps and threats. We can analyze comparing with market competitors, eco-systems, and opensource communities, to understand where we are and what are the gaps.

  • Min-Max analysis. A gap maps to a problem. We may not solve every problem, but we should always be able to plot it on a continuum axis, and scope it with what we are the min-max. We can plot the theory limits vs where we are to find the gaps. We can plot the reaching scope by today’s ability vs in following future, to identify what we should reach but didn’t, what we cannot, and what to prepare in future.

  • Grid search. Continued from Min-Max, we can unfold the properties we care into multi-dimensions of structured space. Use an exhaustive method to divide and categorize the space, and conduct the search. This also helps us find and look into possibilities. In another word, we should have the map, even we cannot reach every location, and we should plot grid axes on the map.

Problems & missing parts in engineering daily work are source of innovation.

  • Online service and operation issues. Apply systematical thinking and in depth. An issue to happen usually reflex missing functionalities, gaps to fill, and opportunities for paradigm shift. “5 whys” is useful method from Critical Thinking, e.g. why the failure is not detected before impact, why auto mitigation not working, why it is captured now rather than at an earlier stage, etc.

  • Don’t repeat yourself. If we are working with repeated issues, with repeated manual operations, with repeated coding, that usually identifies tooling innovations, paradigm shift how we work, or opportunities to change how we share and reuse.

  • Frequently we meet that, want to do A, found A needs B1, B2, B3 first. And then Bi needs again Ci1, Ci2, Ci3 first. This is work blocking but chances for innovation. Think in higher level and abstract general, this usually brings new components to address general needs. Think about the history of NASA and Internet from military.

  • The preset for innovation is to measure it first. Not only abundant metrics, logging, analytics sources help troubleshooting, and help identify the gaps we have. A better measure in finer grain usually reveal what we don’t know before, and breaks the wrong hidden assumptions. These then become new places to improve.

There are wiser views to manage product and scope to help innovation

  • Instead of only focusing on own product, we should reach into the customers about how they use and think. This helps identify the real problems to innovate from.

  • Instead of only focusing on own product, we should extend the scope to understand and learn from surrounding products and teams. There can be things we missed; there can things we can leverage.

  • Keep updated with new feature or services released. The surrounding team may recently build new technologies, added new features, or running new services. We can think how to leverage them to improve own product. We can think how to apply them to new problems and new scopes. This are often sources of innovation, and is well welcomed by other teams.

  • Find better alternatives and improvements. When we face the existing features surrounding us, we can routinely think are there better ways, or can they be improved? Such thinking also generates innovation. Also, review our product, where we are internally & externally, in a holistic view, are thought training and knowledge accumulate that make innovation more fruitful.

Team working is important for innovation

  • It’s non-trival work to keep updated and follow up all technology directions. Build a team, where each member covers a different direction, and do regular share sessions like seminars. It will more efficient.

  • Teamwork helps inspire new ideas, e.g. brainstorm is not running on individual. Ideas passing back-forth, listened and reflected, can usually generate much more.

  • Idea polishing needs teamwork. A new idea needs to be verified. It needs to be extended. It needs to be patched for weakness. Sometime it needs to be turned into another way, or brought into a new level. This usually happen across a team, which idea passing back-forth, interactively, challenge and get challenged. Everyone has the unique experience to help idea grow, to avoid it stuck in a deadend.

  • Besides working in office, innovation also needs info feed from the external world. We can participant regularly in meetups, summits, etc, to meet guys from different places and how they solve related problems. This is usually key to know our industry position, catch up and ensure we are not hitting in a wrong direction.

  • Help other people, and answer questions. Besides providing knowledge, answer questions provide chance to review our knowledge and product again, and understand how people see it from another perspective. This can usually find new gaps. And it broadens our scope and helps other people grow, eventually a positive feed for innovation.

Visions & strategic thinking that help innovation

  • Innovation can be inspired from vision thinking, e.g., what are we doing for the next 1~3 years, and 5~10 years, what will our product be like, what are the technology directions and our position, internally or externally.

  • In another perspective, in the next 3~5 years, how we will be doing daily work and organize the team. What are the key driving factors behind, the bottleneck, and the growth space.

  • What is the lifecycle and position of our product, our technology domain, and our market. Innovation is generally having more opportunities in the emerging new domains, more challenging areas, and generally less easier in the mature and fading markets.

  • Read in a systematically way and compare. It’s comparing rather than linear reading that reveals intrinsics. Innovation comes from the compare

  • Following industry trends, watch big market vendor/leader moves, study venture capital news. This helps find emerging but proven technologies on the climbing curve.

In the end, here attaches the storage driving factors I summarized in scratch, which can help plot the innovation landscape.

1. Hardware
    1. new type of hardware, e.g. 3D-XPoint, SSD, SMR, ZNS, NVMoF
    2. growing capacity of hardware, e.g. TB memory, 1 machine 100+ HDD, manycore, RDMA
    3. changing proportion of hardware, e.g. colocating compute/storage vs disaggregated design Snowflake, faster networking / faster storage device vs not-catching up CPU
    4. smart hardware, e.g. FPGA, Computational Storage, AWS Nitro, Azure Catapult
    5. GPU, DPU, IPU

2. Growing scale of data capacity and IOPS
    1. distributed scale-out storage, Ceph, now everything
    2. distributed filesystem, CephFS, EMC Isilon, Azure HDFS/DataLake
    3. cloud opening 1000+ datacenters worldwide, interconnect, geo-distribution
    4. performance deterministic, SLA

3. COGS saving
    1. ErasureCoding, performance optimization, Kernel Passthrough
    2. Data deduplication
    3. Data tiering, migrating, caching, etc
    4. underlying data representation, data format, physical layout

4. Reliability
    1. Data backup, Copy data management

5. Related technology
    1. container, cloud
    2. TLA+, formal verification
    3. consistent hashing, distributed transactions, append-only LSM tree, ART / Mass-tree indexes

6. User interface changing, new business models
    1. SQL, no SQL, new SQL
    2. OLAP, OLTP, HTAP
    3. in-memory storage, all flash, NVMe storage
    4. Cloud, Hybrid Cloud, Cloud offloading, Hyper-converged, IoT, Edge computing, Blockchain
    5. many pushing from new changing consumer, e.g. the active Database community, VM & virtualization, NFS, page & block storage, archiving, data backup, AI & machine learning & deep learning, GPU, Datalake & enterprise data management and analysis, software defined DC/storage, containerization, etc
    5. stream processing, transactional streaming, evolving table, RDD, XOR linage
    6. video processing. online realtime podcasting
    
7. Co-design with surrounding
    1. Co-design with database some years, and then decouple design some years
    2. Co-design the custom hardware some years, and then decouple with whitebox commodity hardware some years
    3. Open-Channel SSD, ZNS GC, SMR GC

8. Security
    1. Zero trust, confidential computing
    2. Ransomware, Immutable Storage

9. Data management
    1. Operational model changing, Orchestration
    2. on-premise and hybrid cloud management and COGS improvement, metrics and analytics COGS saving
    3. data migration, shipping, Edge, caching, Tiering

Some tips for personal lifestyle

  • Time to stay alone uninterrupted is extremely important. It triggers active mind thoughtss when being idle, or do routine hand work (non-mental involving), or casual reading.

  • Don’t get too busy. Free relaxed time is important. Sleep more than enough to let your mind reorganize itself. To have a breakthrough idea, it doesn’t take time, it takes space.

  • Phone browsing gets updates but is time consuming. Separate your time to fragmented time and big block time. Only phone with fragmented time. Big block time should do big block things.

  • Changing the living environment helps. Stay at a new place for a few days to get rid of stereotypes or fixed thinking mindsets.

Useful tips from more sources

  • Learn from history, understand upstream and downstream, and stand on shoulders of giants. Summarize and analyze down experiences and issues systematically (归纳和总结). Dig to bottom rather than solve case by case (治本而不是治标). 局部、全局,以及发散的眼光,前瞻未来的问题。 发现问题、眼光,定义问题。 Architecture Thinkings by 阿里道延

  • Practice writing and presenting to others. Clarify and make sure you only present what you understand, and make sure the audience can understand. Writing means clarifying your mind; Always try to write something – some of the best ideas will come to you by writing. Instructions for PhD Students - Dimitris Papadias, HKUST



Create an Issue or comment below