Openstack Tokyo Summit Notes

Openstack Summit Keynote:

    Jonathan Bryce Exec director @ Openstack Foundation: General Introduction
        Openstack now has Certificate for Openstack Administrator. You can attend the test.
        Introduction to Openstack development model vs common enterprise projects
        Introduction to Openstack sub-projects
        How to get involved in Def Core: 1) by providing data 2) participant in Def Core meeting
        Openstack adoption in market. The picture is each sub-project level, e.g. Magnum adoption rate.
        New Openstack project status page released. See each sub-project maturity status. Check it at: http://www.openstack.org/software/project-navigator
        Openstack is powerful vs docker/k8s because by one platform you have all models: VM, baremetal, container on VM, container on baremetal
        Federated Identity released in Kilo and keep being contributed in Liberty
        Super User awards: winner is NTT Group. See who is greatly using Openstack here.
        
    Lachlan Evenson Cloud Platform Engineering Team Lead @ Lithium Technologies: Use container on Openstack to quickly deploy infra with no engineering effort
        Be able to deploy container cloud on Openstack in less than 2 weeks. Awesome demo pages. All in pods ready for you.
        Kubot. A chat robot who you can ask to operate k8s, e.g. scale out.
        Demo the change code and quickly re-deploy the crocodile game
        
    Takuya Ito Sr. Manager of Infra Engineering and Openstack Blackbelt @ Openstack use case in Yahoo JP
        The scale: 61B page view per month. 20PB in data storage. 20+ clusters operating. 50000+ instances.
        The crazy workload spike (+300%) when a nature disaster happens. We have mission-critical applications on Openstack.
        Our mission is to make abstraction of the datacenter, with Openstack as the core. Common APIs are important.
        
Shingled Magnetic Recording (SMR) Drivers and Swift Object Storage

    Various tests are carried out on SMR. Use a Swift Simulator to understand how SMR performances.
        Increase the concurrency and workloads until the performance deteriorates to be unusable. 
        Drive managed vs Host managed
        Write vs Write-Heavy Read/Write vs Read.
        SMR Drive vs PMR Direct Access vs PMR No Direct Access
        Many other tests ... SMR is not performing well vs PMR
    
    Should we deploy SMR on Swift?
        You should not use SMR Driver on Swift. It brings performance drops, especially for small objects. It doesn't worth the storage density gained.
        Don't use SMR for general purpose. Use it for Swift with large object or very large bandwidth ingest.
        Good for use cases such as video surveillance cases, DNA analytics
    
    Future work
        Make it more configurable for the operator.
        SMR native filesystems. Teach Swift to speak SMR.
    
    Good questions audience asked
        Cache tiering with SMR? No such much related. We could use it if we implement tiering which supports SMR.
        Host aware drivers? Not quite available now. There are papers and academic discussions.
        
Full Stack Application Security in the Cloud

    The speaker didn't show up, the speech magically disappeared and jumped to the next one.

Running Openstack for Several Years and Live to Tell the Tale: by Redhat

    Undercloud
        RHEL + TripleO deployed via Kilo
        Ironic
            benchmarking you hardware is highly recommended
            Upload hardware profile to Swift
        Heat
            customize TripleO template to deploy Overcloud
            Configure Overcloud with PUppet

    Reference Architecture
        Controller 3+2n, Compute X, Storage X nodes
        Network isolation: provisioning/mgmt, internal API, tenant networks, storage, storage mgmt, external API & floating IP
    
    Logging
        Centralized logging by Fluentd + Kibana + Elasticsearch
         
    Monitoring
        Sensu with Uchiwa dashboard
        System checks + Openstack checks. A lot of them.
        
    Backup
        Run from external server.
        Only backup Ceph volumes
        Full and incremental
     
    Tips and Tricks
        Sync time with NTP across your servers
        Network MTU 9000 if hardware supports jumbo frame. Disable TSO, GSO. Disable rp_filter. Disable GRO, GSO, LRO.
        HAProxy increase maxconn. Increase Galera timeout. Increase RabbitMQ timeout.
        RabbitMQ limits: `rabbitmqctl status | grep file_descriptors -A4` increase it
        Set rabbit_durable_queue. Set rabbit_ha_quque. Set expire policy to avoid amount of orphans queue.
        MySQL increase open_files_limit, LimitNOFILE, max_connections. Monitor the number of active connections
        And more. Check the published video on Youtube
        
Hyperconverged for Openstack environments, does it make sense

    Convergence is the third wave of human evolving. Multi-vendor converged; single vendor converged; Hyper converged. 
    
    From software defined infra to webscale IT
    
    features of 3rd wave convergence.
        Efficient data management
        Infra convergence
        Scalability
        VM Centricity: primary focus on the virtualized workloads, no baremetal
        Unified management
    
    Customer space of where hyperconverged models emerge
        Hyperscalers / Web giants
        Service provider - Telecoms
        Managed Openstack
        Distros
        DIY
    
Multisite Openstack - Deep Dive

    Even the doorway is packed up with people. Highly recommend this session.
    When try to connect the network between two sites, the IPs, MACs and router tables may change after VM migration. They have implemented the cross-site router to handle it.
    Cross-site firewall is relatively easy to implement. They also implement the cross-site loadbalancer.
    Router peering creates a link between two sites. L2 and L3 network connectivity involves a lot work.

The Container Ecosystem, the Openstack Magnum Project, the Open Container Initiative, and You: by IBM

    Containers isolate the process resources, share host kernel and avoid hardware emulation, package app dependencies, easy to run and move. But there are cost
    Containers are not new, it starts from FreeBSD Jail, to Linux-VServer, to Solaries Zones.
    Existing container efforts in Openstack: Nova docker driver, a plugin for Heat to orchestrate docker resources, Kolla containerized Openstack services, Murano market place catelog, Magnum to manage CaaS

    Magnum implements OCI & CNCF.
        Introduction to Magnum team history, status and architecture
        Introduction Linux Foundation Open Container Initiative (OCI). OCI is lightweight. OCI is not bound to higher level constructs such as particular client or orchestration. OCI aims to meld Docker vs CoreOS Rocket. OCI specifies OCF (Open Container Format).
        Introduction of the Cloud Native Computing Foundation (CNCF). Container packaged, dynamically managed, micro-service focused. OCI specifies container format. CNCF specifies container platform common parts. OCI targets container image portability. CNCF targets container application portability.
        Magnum implements OCI & CNCF. You can see how Magnum positions itself now.

Openstack Summit Keynote

    Neutron is the most actively developed project in Openstack, about 30% growth of production adoption last year. SDN market grows fast.
    Neutron functional decomposed: virtual port, network, subnet, binding; LBaaS, VPNaaS, FWaaS; L2GW, SFC, BPG-MPLS VPN.
    Kuryr (sounds like 'career'): Container network by Neutron, leverging libnetwork. Key project this year. 
    
    Rackspace releases Carina (free beta), an instant-on container native environment, getcarina.com. Now we have container based public cloud :-)
    SK Telecom is demonstrating their 5G mobile network, 100x-1000x speed up than 4G. SDN & NFV are built on Openstack. Pilot release should be available at 2018.

Manila Work Sessions

    We have attended the Manila work sessions. It is a cosy small room on the ground floor with 10 - 15 core developers participating. A long table is put in the middle of the room, members sitting around two sides. The PTL sits in the head position and operates the screen. The conversation is led by PTL. There are many topics being discussed. The meeting is quite cold actually. For each topic only a few members who are related responses. I guess that Manila core developers are divided and each of them focus on different parts. The meeting results are recorded on Google share doc. It will be published after meeting.
    
    I guess this part of summit is seldom published on Youtube. Only results are shown on Etherpads. If not attend, you lose chance to decide future Openstack.
    
    The discussion is very detailed into Manila. There are common Manila operations listed on the doc. Network sharing issues are discussed. Then access list and rules and batch update or not are discussed. Manila holds the identity of the filesystems created. Ceph has its native authentication methods such as cephx or kerberos. They have some conflicts between each other and that's what is discussed later. How guest VM should access storage network is an problem. For Ceph, the two networks mon and osd are separated. Guest may not be able to access mon to get volume information, even if it has access to osd network.
    
    About how Cinder CI is used to test vendor storages. The vendor subscribes code patch notification. Once a code patch is submitted, the vendor system pulls it down, spin up devstacks to test it out. In most cases devstack single node is used. There may be a set of devstack env for each type of stroage being tested. Devstack often provokes problems because of its high pace of development and unstability.
    
Optimizing and Extending Overlay Network for Containers

    Experiments and the results (with numbers and graphs) to show container overlay network kills performance

        Env setup
            3 nova instances connected by eth0, one of which is kube-master. There is DNS service on the private subnet. Load balancers are connected to the subnet. There is also an router interface and router connected to the subnet. The router provides floating IP. The overlay network needed for k8s is provided by Flannel. 1 kube minion on the same host with kube master, another on the different one.
        
        Single overlay kills performance
            From server to vm-flat: bw (bandwidth) to 82%, latency to 350%
            From vm-flat to vm-vlan: bw to 96%, latency to 114%
            From vm-flat to vm-overlay: bw to 26%, latency to 108%
            Change MTU from 1450 to 1000, the bw decrease to 50%.
            
        Double overlay kills twice
            Throughput drops 41%, latency increase 34%
            
        Flannel UDP and VXLAN backend
            VXLAN obtains 3-5x throughput
            
    Pluggable libnetwork drivers: Null, Host, Bridge, Overlay, Remote. The 'Remote' one uses plugin to communicate with remote provide. It can be utilized by Neutron. Kuryr is a Docker network plugin that uses Neutron. Docker network concepts are mapped to Neutron (Docker IPAM is still under work). With Kuryr we can avoid overlay on top of overlay. Neutron VLAN-Aware VMs have initial patches under review; the trunk port on VM connects multiple containers in the VM.
    
Ceph Community Talk on High-Performance Solid State Ceph

    Ceph has been greatly improved from Dumpling, both performance and maturity. Reduce avg latency and spike. 95th+ percentile starts to exceeds 20ms. Venders are contributing and optimizing Ceph for their devices.
    
    Intel involves a lot in Ceph community contribution. The first Intel Ceph Hackathon focus on performance optimization. Intel also donated 8 high performance hardware nodes. Focus areas of Intel Ceph optimization includes PMStore (persistent memory store), client-side caching enhancement, lockless C++ wrapper classes, RBD & RADOS data-path optimization, cache tiering optimization.

    Samsung contributes to Ceph. For SSD interface improvements, existing read path is synchronized in OSD layer. It is extended to support async read. For messenger performance enhancements, the new XIO Messenger is still experimental. XIO is extended to support RDMA NIC ports.
    
    Sandisk contributes to Ceph. About OSD optimizations, in the SSD context, CPU becomes bottle neck again. It needs more parallel and less CPU/OPs. There are many iterative small improvements in OSD read path. The existing write path strategy is for HDD, Sandisk modified buffering/writing strategy for SSD. Future there are RDMA intra-cluster communications, NewStore that reduces write amplification, and improved memory allocation.
    
    WAL is widely used in Ceph, but in SSD context, writing everything twice is inefficient. Essentially we write WAL + actual data because we need to relocate data. But SSD FTL actually already manages data relocating. NewStore may achieve writing data only once (except metadata).

Kubernetes Cluster on Openstack
    
    This is an introduction level talk for k8s.
    
    Why Kubernetes?
        Abstract the application with Pods.
        Ability to group Pods using labels.
        Enable accessing to Pod group using service abstraction.
        Pod management, self-healing.
    
    Details
        K8s architecture.
        K8s network model.
        Issue of docker default networking model with k8s. To solve, opt1: routing for Pod networks; opt2, use overlay network.
    
    I asked
        K8s use flannel for underlying overlay network. How can I build complex network model like Neutron (seperated networks, routers)? No much idea yet. But be ware of overlay network performance hit.
        If I use multiple redis nodes and app uses consistent hashing to access, the k8s service model breaks the paradigm. How to solve? No much idea yet. A walkaround is to get IP of each pod and access them.
    
Building Clouds for the Financial Industry: Challenges and Solutions

    ShenZhen Securities Clearing Corporation .Ltd (SSCC) is building an IaaS, PaaS, and SaaS platform for their financial customers, new datacenter at 东莞. Cloud enables the user stories. Built by Mirantis + Openstack.
    
    User stories
        User case 1: Market data cloud
        Use Case 2: Fund Cloud
        Use Case 3: Face recognition cloud
        Independent software vendors: the partner companies.
    
    Why Openstack?
        No vendor lock-in
        Customization
        Independent control
        Openstack Merits: Scalability and performance, good API interface, quick update speed
    
    Mirantis Partner
        Using KVM.
        Using Ceph as Cinder backend.
        Mirantis provided SDN solution, Juniper Contrail.
        Use Murano to deploy. Murano is the financial application catalog.
        Use Sahara for big data. Special host aggregates for Hadoop related.
        LMA - Logging, Monitoring, Alerting. Elasticsearch, InfluxDB, Grafana.
    
    Juniper Contrail as the selected SDN
        HA in control and data planes.
        Horizontally scalable L3
        Service chaining to provide secure services (vDPI, vFW)
        No hardware lock-in
        Integrated with Fuel plugin with full automation
        
    Storage design
        Ceph SAS pool: most storage needs and Nova ephemeral storage
        Ceph SSD pool: high disk IO workloads
        SAN Storage pool: mission critical workload
        
    The Openstack is already online and real customers are using it, but only business financial institutes (主要是中小的金融机构系统)。Currently the cloud is provided via SaaS to its consumers. Later when it gets more mature, larger financial institutes may join and consume it by hybrid cloud model.
    
    How does the financial cloud differs from a common cloud? High priority for security and stability. The cost is not so much concerned. Service is provided by SaaS.
    
The state of Ceph, Manila, and Containers in Openstack: by Sage Weil

    Why file? Why not block? 
        Container volume are FS.
        App uses FS first.
        Ext4 etc FS are exclusive access.
        Block volume size expands without administrator control.
    Ceph has multiple clients: Linux kernel, ceph-fuse, libcephfs.so (Samba, Ganesha, Hadoop)
    The key feature of CephFS: dynamic metadata partitioning
    CephFS road to production: "Production-ready" Ceph FS at Jewel release (Q1 2016)
    FSCK and repair: a lot of tools. Tool must be available for CephFS to go production.
    
    Manila File Storage awkwards
        Manila also manages part of connectivity problems, manage "share networks" via Neutron.
        User must attach guest to share network, user must mount the shared.
    
    Genesha + libcphefs model for CephFS via Manila.
        Expensive: Extra hop. Extra VM.
        It is not HA.
    Native Ceph Driver for Manila
        Client needs modern kernel.
        Comming soon.
    cephfs-volume-manage.py -> libcephfs.py -> libcephfs.so -> CephFS
    Security issues
        Tenants have access to storage network. CephFS has to ensure tenant isolation.
        New CephFS path-based authentication.
        Ceph's security becomes the only barrier.
    
    KVM + 9P/VirtFS + libcephfs.so model for CephFS via Manila
        Tenants remain isolated from storage layer.
        More compact
        Prototyping by Jevon Qioa, Haomai Wang, etc
    
    KVM + NFS + NFSD/GANESHA + CephFS
        Tenants remain isolated from storage layer.
        More compact
        NFS to Host: Problems with TCP/IP. Slightly awkward network setup.
        Cons on the ppt. There are Cons.    
        AF_VSOCK
            VMware vSocket. Zero configuration.
        NFS to Host: VSock (Community learned from VMware AF_SOCK?)
            NFS 4.1 only.
            Various patches are under review.
    
    KVM + NFS (VSock) + NFSD + CephFS
        We like the VSock-based model
    
    I definitely want the ppt because a lot of detailed are written on the ppt but Sage goes too fast.
    
Storage Vendors are Killing Cinder: by Rackspace

    The title is bluffing. They showed Lunr & Lunrdriver, a Rackspace homebrew LVM backend for Cinder.
    
    Core conflicts between Cinder and its vendors are
        DefCore wants uniformity
        Vendors want differentiation
        Openstack wants it all
        How to make it inter-operable?
        
    One of the audience suggests we should add a do-nothing-driver for Cinder. Because some customer just want to disable Cinder, and manage volumes from outside.

My Quick Cap

    From the half year last cycle, Neutron added IPAM (IP address management), L2GW (first seen in Kilo), SFC (service function chaining), BPG-MPLS VPN, Kuryr (solve container overlay problem via libnetwork), Service Chaining (still blueprint but implemented in distributors. same stuff with SFC?). Ceph added PMStore (persistent memory store), RDMA (remote DMA) support, XIO Messenger, many OSD RW path optimize. I would say this year is a network year for Openstack.
    
    Materials I found to understand the new sparkling words
    
        Neutron IPAM (vs Docker IPAM)
                http://specs.openstack.org/openstack/neutron-specs/specs/kilo/neutron-ipam.html
                https://wiki.openstack.org/wiki/Blueprint-ipam-extensions-for-neutron
                https://blueprints.launchpad.net/neutron/+spec/neutron-ipam

        Neutron L2GW
            L2GW: https://wiki.openstack.org/wiki/Neutron/L2-GW
            http://docs.openstack.org/developer/networking-l2gw/usage.html
            what is service plugin?
                https://wiki.openstack.org/wiki/Neutron/ServiceTypeFramework
                https://github.com/openstack/neutron/blob/master/neutron/plugins/common/constants.py
        Neutron SFC
            https://wiki.openstack.org/wiki/Neutron/ServiceInsertionAndChaining
        Neutron BPG-MPLS VPN
            https://wiki.openstack.org/wiki/Neutron/MPLSVPNaaS
            https://wiki.openstack.org/wiki/Neutron/DynamicRouting/UseCases
            https://wiki.openstack.org/wiki/Neutron/DynamicRouting/TestingDynamicRouting
            what is Quagga?
                https://en.wikipedia.org/wiki/Quagga_(software)
                http://www.slideshare.net/KeithTobin1/architecting

        Kuryr arch. how it works
            https://github.com/openstack/kuryr/blob/master/doc/source/devref/goals_and_use_cases.rst or http://docs.openstack.org/developer/kuryr/devref/goals_and_use_cases.html
            https://etherpad.openstack.org/p/magnum-kuryr
            https://github.com/openstack/kuryr/blob/master/doc/source/devref/libnetwork_remote_driver_design.rst or http://docs.openstack.org/developer/kuryr/devref/libnetwork_remote_driver_design.html#libnetwork-user-workflow-with-kuryr-as-remove-driver-host-networking
            http://superuser.openstack.org/articles/project-kuryr-brings-container-networking-to-openstack-neutron
            https://wiki.openstack.org/wiki/Meetings/Kuryr
            http://www.slideshare.net/takufukushima79/container-orchestration-integration-openstack-kuryr
            http://blog.imaginea.com/cutting-edge-openstack-adding-container-support-to-iaas/

        ceph PMStore
            http://tracker.ceph.com/projects/ceph/wiki/PMStore_-_new_OSD_backend
            https://www.youtube.com/watch?v=Oy1xonZk20U
            http://tracker.ceph.com/projects/ceph/wiki/Hadoop_over_Ceph_RGW_status_update and https://www.youtube.com/watch?v=oqO_psxwJFk
            http://tracker.ceph.com/projects/ceph/wiki/Ceph_0_day_for_performance_regression and https://www.youtube.com/watch?v=t0A8syTfaY0 and https://lwn.net/Articles/514278/


        ceph RDMA NIC ports & ceph XIO Messenger
            http://tracker.ceph.com/projects/ceph/wiki/Accelio_xio_integration_with_kernel_RBD_client_for_RDMA_support
            http://tracker.ceph.com/projects/ceph/wiki/Accelio_RDMA_Messenger
            http://www.slideshare.net/mellanox/mellanox-high-performance-networks-for-ceph
                1. Mellanox networking + accelio RDMA
            http://www.slideshare.net/somnathroy7568/ceph-on-rdma
            https://community.mellanox.com/docs/DOC-2141
            http://www.snia.org/sites/default/files/JohnKim_CephWithHighPerformanceNetworks_V2.pdf
            about accelio: http://www.accelio.org/wp-content/themes/pyramid_child/pdf/WP_Accelio_OpenSource_IO_Message_and_RPC_Acceleration_Library.pdf

        SDN service chaining
            http://packetpushers.net/service-chaining/
            http://www.sdnzone.com/topics/software-defined-network/articles/362831-service-chaining-seems-important-but-what-it-aga.htm
            http://www.tail-f.com/service-chaining-with-sdn-and-openflow-controllers/
            https://blueprints.launchpad.net/neutron/+spec/openstack-service-chain-framework
            https://review.openstack.org/#/c/93524/13/specs/juno/service-chaining.rst
            https://blueprints.launchpad.net/neutron/+spec/neutron-api-extension-for-service-chaining

        libcephfs.so & hadoop
            http://noahdesu.github.io/2015/07/12/hadoop-ceph-diving-in.html

Why Reinvent the Wheel? Using Murano, Heat, Container Clusting & Ceilometer to Provide Auto-scaling & Enforce Self-healing Best Practices in Applications: by Mirantis

    If security is more important, run container in VM. If density is more important, run container in baremetal. K8s, Mesos, Docker Swarm & Compose are supported.
    How to make container infra reliable & scalable? Deploy & Scale by: Murano & Magnum. Mointoring by Ceilometer & Zabbix
    Self-healing capabilities are provided by Murano. Application workflow are callable by API (imperative workflow language). Workflow is used for removing failed node or create new node. I was thinking application deployment should be represented as an object language, with common operation exposed.

Leveraging Kubernetes to Scale Containers on Hybrid Multi-cloud Cluster: by Mirantis

    What? Mirantis Again? 标题党? This is only early work.
    Use Murano + K8s for auto-scaling.
    Where is the hybrid multi-cloud?

NFV Orchestration - Go Beyond Virtualization

    The room is totally. Even doorway is packed with people. I never managed to get in. Highly recommend we take a look later on Youtube.
    
Decomposing Lithium's Monolith with Kubernetes and Openstack

    Lithium is the company name. Monolith is the big monolith app.
    Some app is designed to work on VM. It is very hard to bring it on container.
    Container offered simplified packaging and deployment to the cloud. Need developer-led.
    Should you split the monolith? The monolith is long running, 50 years. It is hard to split it up. For deployment docker pipeline can be used. Be incremental.
    You can't containerize everything. Not just for "stateless" web front-ends.
    Why k8s? little engineering effort, docker primitives, openstack provides the platform and fill the gap (e.g. cinder storage). Cross-platform support, AWS CloudCoreo, k8s can be deployed on AWS in the same way as local.
    Containers force you to rethink everything: logging, monitoring, secret management, config management, try not to create container anti-patterns (e.g. ssh into container; multiple process in a container)
    
    The results
        Time spent on infra vs features
        single automatable pattern including CI/CD
        Infra tools cloud follow the same pattern
        AWS even acknowledged the problem
        
        Higher code coverage
        Smaller 
        Complex deployment options have been simplified
        Canary releases and rolling-upgrades and rollbacks
    
    I asked how app can access multiple ip of redis via service model on k8s. Answer: use endpoints api (http://kubernetes.io/v1.0/docs/user-guide/services.html). In the doc, I think headless service & service discovery is also the solution (http://kubernetes.io/v1.0/docs/user-guide/services.html#headless-services).

Turn Openstack into the Global InterCloud - Now!
    
    It's a bombastic title -_-o Talk is about future. The speaker looks like a scientist from Aerospace.
    IaaS-level federation, SaaS-level federation, App-level federation. Keystone federation is here already. Federation agent is the thing that manages local user & resource info.
    Federation deploy models: Pair federations, hierarchical federations, centralized 3rd-party federation, peer-to-peer federation, ..., interclouds.
    What does federation actually require?
        User Authentication
        Federation Discovery
        Interoperability
        Membership, governance and trust
        Federated identity management
        Federated resource access
    Trust federation - a precedent: www.igtf.net. Global grid enabled by trust protocol.
    A KeyVOMS for secure service discovery.
    Keystone is very close to what's needed for general federation management.
    
Scalable and Reliable Openstack Deployments on FlexPod with Red Hat Openstack Platform: by Cisco & NetApp

    The companies names are interesting.
    FlexPox is a physical box based on Cisco UCS, Cisco Fabric interconnect and NetApp storage, Nexus 1000V, Hypervisor of KVM or Microsoft or VMware. Redhat-OSP installer with NetApp integration. Local image caching by NetApp FlexVol. Nexus 1000v is configured once VM boot up.
    CVD Design: http://bit.lv/1LFCHEz
    
Storage is not Virtualized Enough: by Huawei Zhipeng Huang
    
    Inspired by NFS, Storage Function Virtualization (SFV).
    SFV in Openstack. Several BP submitted.    
    Google group 'sfv-dev'
    Difference from Ceph (SDS): SFV tries to run Ceph inside VM, with no performance penalty. Service Chaining can also be borrowed from SDN / NFV.
    
Unraveling Docker Security: Lessons Learned from a Production Cloud: by IBM Research & IBM Cloud
    
    Threat model
        Containers attack on other containers running on the same machine. E.g. see which process are running, which files are used, which network stack, what hostname, IPC other containers are using.
        Containers attach on host machine. E.g. Misconfigured containers, malicious containers, is root inside a container also root outside (they share the same kernel)? are cpu/mem/disk limit obeyed? can a container gain privileged capabilities? are other limits obeyed (e.g. fork())? can container mount of DoS host filesystem?
        Attacks launched from public internet. E.g. Scan open ports, ...
    
    Protections
        User namespaces, cgroups, Linux capabilities, Linux security modules AppArmor/SELinux, Seccomp, Restrict Docker API, Docker and storage engine configurations (TLS, nproc/fd limits, docker security checklist, make fs readonly, files quota), Docker Registry Security (V2 API: content is addressable cryptographic hash, signing & verification, digests & manifests, safe distribution over untrust channel, naming and content are separated)
        Use trust computing and TPM for host integration verification and VT-d for better isolation
        
    Possible attacks on Containers
        Forkbomb. DoS on host. Resource exhaustion on host storage. App level vulnerabilities (e.g. weak credential, password in dockerfile). 

    I recommend we get the ppt because many security checklists are listed there.

NFV Bakeoff Panel
    
    This room is packed with people, but not doorway this time. People love NFV this summit.
    
    Challenges why NFV has not been widely deployed in production
        mission critical app?, really cut down cost?, converge with openstack platform, poc after poc after poc,
        automation & orchestration beyond just virtualization, standard & vendor interoperable, standards are not mature and issue prone,
        interoperable with existing systems, need to talk with existing technologies, in openstack there are always more than one way to do anything and you have to decide which to use (Murano? Heat?)
        same level of performance and functionality (e.g. firewalls), initial steps in orchestration,
        
    What is the missing capabilities?
        orchestration, interoperate with nova compute, make features easy to consume for operators, plug'n'play,
        more works for securities, standardize how things communicate, Heat / Murano co-works, BGP & VPN projects,
        do move into an API driven system, steady env that operators need, more work in nova and neutron (e.g. scheduler),
        need SDN for NFV to work, need neutron support,
        
    Audience: Current docker adoption and openstack and five 9?
        Openstack needs to be HA. Protect the openstack. Neutron HA. More issues should be expected. Keep the pace in Openstack and OpenNFV.
    
    Tacker Project: https://wiki.openstack.org/wiki/Tacker
    OpenNFV Project: https://www.opnfv.org/
    
Block Storage Replication with Cinder: by John Griffith & Ed Balduf, both from Solidfire

    Purpose: DR for non-cloud apps (traditional app). NEVER - 2 different vendor backends. Use cases        
        2 backends, same cloud
        2 backends, different cloud
        2 backends, one not in a cloud
        Replication one to multiple backends
        Automated -vs- Non-automated failover; Consistency group recover
        Snapshot replication -vs- continuous replication
    Volume replication is admin only, and invisible to normal user.
    
    There are a lot of difficulties how we got there. Looks like dev progress Cinder replication is kind of stuck. 
        many voices lack agree, unique characteristics that don't translate all, lack of testability, rush at the end of release cycle, patch reviewed many many times, etc. (Vendors are killing Cinder? again? -_-o). Kind of stuck. 
        V1 lessons learned (V1 semantics are no longer used): Lack of community involvement, only worked for one vendor, the only 1 person who understand it in depth is no longer available now.
        The tests and CI tests are useful but they all go on 1 node openstack, so no way to actually test the replication. So the current status of community is zero testing.
        V2 lessons learned: heavy involvement from multiple vendors, reviews reviews reviews, everything stuck on spec review, DON'T RUSH!!!, we will sell no wine before it's time. There is a lot of talk in Cinder about replication, but no implementations in Liberty (really?).
        
    Demo (no live demo because not fully implemented). PPT walkthrough. Then there is a recording of demo.
        Config file: `replication_device=xxx`
        Volume-types: replication_enabled, replication_type, replication_count
        Terms: enabled/disabled, status active,error,inactive,disabled, tasks replication enable/disable/failover, list replication targets.
    
    Drivers in work (or done): solidfire, pure storage, HP, IBM. Drivers still in progress (under review): Dell, EMC (VNX), NTAP (pool level)
        
雲泥の差 'The Separation Between Clouds and Mud' - Operating OpenStack Private Clouds: by Rackspace

    Speakers are US/Euro. Where is the Japanese?

    Approach to Product Management: the key is to operate the cloud, not just build it for customer. The "customer" is a group of operators who provide cloud service to their customer.
    
    Design Principles
        Focus on what "Day 2" looks like, or Day 20, or Day 400
        "Opinionated Deployment" - design the deployment layout for customer needs
        iterate -> learn -> iterate -> learn -> iterate
    
    The Virtuous Cycle
        Openstack grows. The growth drives business.
        Look into the community to find what new features are enabled.
        Document and training.
        Feedback from customer builds up product.
        Community + Customer + Operators + PM & Engineers feedback adds to product cycle
    
    Net Promoter Score: NPS-R, NPS-T
        a. How are we doing?
        b. Would you recommend?
        c. Rating and free text
        
    Operations - Social Contract: Engage, Lead, Operate, Communicate

    Recognize the difference between deploying and operating a cloud: love the loop, create & define team identity & culture, balance process and empowerment

Trusted and Secure Containers for Enterprise Deployment: by Intel

    Trusted VMs and Trusted Containers.
    
    To define the container security problem
        Docker's host integrity
        Docker container integrity verification
        Runtime protection of container & enhanced isolation
        Intelligence orchestration - Openstack as singular control panel.

    To solve
        Ensure container are launched on Trusted Docker Hosts:
            Boot-time integrity: Measured Launch of Boot Process and components by Intel TXT
            Docker daemon and associated components are added to TCB and Measured
            Chain of trust: H/W -> FW -> BIOS -> OS -> Docker Engine
            Remote Attestation using Attestation Authority
        Ensure docker images are not tampered prior to launch
            Measure and verify docker images, chain of trust: H/W -> FW -> BIOS -> OS -> Docker Engine -> Docker images layers
            Sign and verify images with hardware supported security: Intel TXT + TPM. Can work with Notary* - Docker Content Trust Model
            Boundary Control/Geo-Tagging for docker runtime compliance
        Leveraging trusted VMs for asserting trust of the host and the VM. Include Docker daemon as part of VMs TCB.
        Who verifies the trust? Intelligent Orchestration (Openstack, Swarm, K8s, Mesos, Fleet). For Openstack
            Nova scheduler selects TPM. Use Trust filter, ImageProperties filter, etc.
            Docker daemon to intercept container launch and request to Measured agent
            Infra change to enable Intel TXT & TPM
    
    Looking ahead: hardware-based runtime integrity
        Intel KGT
        Extend launch time integrity to runtime integrity
        VT-x, xmon, de-privileged OS, monitor/control access to critical assets (CRs, MSRs, Memory Pages, ...)
        Policy describes assets to be monitored and action to be taken
        
    H/W assisted isolaiton: Intel Clear Containers
        use VT-x instead of namespace for isolation between containers (KVM?)
        Provides an extra, rooted-in-hardware security layer for containers
        Still support the advantages of container model
    
    So I think from the last time Intel released Clear Container, now it becomes a complete hardware-assisted container & integrity solution. Intel has done lots of work in hardware-assisted integrity.

Cinder Contributor Meetup (Morning)

    Mitaka Design Summit Etherpads: https://wiki.openstack.org/wiki/Design_Summit/Mitaka/Etherpads. Today's Cinder Etherpad: https://etherpad.openstack.org/p/mitaka-cinder-contributor-meetup. There is recording for this meeting, need to know where it is going to be published. Besides Cinder devs, many vendor representatives are also included in the meeting to be heard from. They also bring their customer use cases. There are also distributor guys here. The meeting is a bit exhausting and I know why everybody carries water now -_-o There are breaking times. People form in groups and discuss various topics. I guess that's another important time where people shape opinions and agreements, to decide the next step Cinder.
    
    Joash from keystone discuss how cinder should work with keystone federation. The volume is attached/detached from another site. VMs can be in this or another site. Keystone federation is working on networking. We want it to work for Nova and Cinder side. Two backends of CG (consistency group) should be the same. CG volumes may have interfere with federation. The datacenters need to be on a shared network for the storage, or at least IP routeable. As the distance increases, latency increases, there can be new race conditions need to tackle with.
    
    The next topic is CG interaction with volume replication. Currently the user can replicate one volume in a CG individually, which he should not be able to do. CG operation process and API should support replication. What's more is that replication is still not fully finished now. We need to inform driver vendors to take CG into consideration when they implement replication. We need to think ahead in case someone is implementing it but we found there is fundamental problem in replication design. Looks like replication is a headache for everyone. Replication can be used for volume, pool, CG, logical pool; that adds complexity. We've been working on Replication V1 (discarded), Replication V2. Now it's Replication V2.1. People keep arguing and just won't reach an agreement. Xing is very active and leading expert for CG & replication to push the progress (but shouldn't there be more persons to help?). Current Cinder implementation allows CG to span across pool, but some vendor may not agree; however replication should be able to span across pool. I found the Keystone guy of the last topic has already disappeared. The vendor can report its capabilities. People are finding ways to to avoid duplicity of CG x Replication x VG x ... Vendors CIs are also going to grow bigger and bigger because these combination scenarios. 
    
    To backup snapshot, currently we can backup volume but not snapshot. We can create an silent volume from the snapshot and reuse volume backup. That's a pattern we should follow.
    
    To improve config file organization, we need to clearly separate per-backend vs per-host vs default section. The default section cascades to driver sections. Maybe we can add warning for misconfigures. Maybe add a 'global' section. Upgrade should be taken into consideration. Which conf option the driver can override from the default section, e.g. oversubscription ratio?
    
    Next we discuss about minimum functions that a driver should support. manage/unmanage is under question because few people really use that. Retype is under questioned because not all vendor supports it and it is not supported in the same way.

    Then, how could Ironic attach a volume without Nova. We now have two source of truth for volume attachment, Nova and Ironic. That's a concern. Deleting a volume from a running instance is dangerous, just like press the power button. Ironic needs an API call that attaches/detaches a volume like Nova. Force-detach may have problems in this context.
    
    The default oversubscription value is 20%. We discuss whether we should change the default value. If changed, we should communicate to driver developers so they can make changes accordingly. Driver conf should be able to override the global oversubscription. Another problem is that if you want to change oversubscription rate, the only option is to change .conf file. You cannot change it on the fly. It is too risky to change the default. Look for <driver>_max_oversubscription_ratio generic option.
    
    Cinder functional tests for python-cinderclient. Reason http://lists.openstack.org/pipermail/openstack-dev/2015-July/068716.html. Neutron has a lot of functional tests implemented and can be used as a good reference. Example https://github.com/openstack/python-cinderclient/tree/master/cinderclient/tests/functional, https://github.com/openstack/neutron/tree/master/neutron/tests/functional.
    
Cinder Contributor Meetup (Afternoon)
    
    The `cinder create 1` the 'Gb' vs 'Gib'? You can change client but don't change underlying api because that will affect everyone. https://en.wikipedia.org/wiki/Gibibyte
    
    Cinder rpc and objects are versioned. An api can switch between new or old versions, so that rolling upgrade can be carried out per component. API needs to be backward compatible. We discuss the possible work items and the versioned api/objects workflow. This is key to enable rolling upgrade and we should make it ready at Mitaka release. Database compaction (compact the db migrations?) also need to be done. Need dedicated core dev to help review these patches because they get merged too slow.
    
    Volume migration improvement. A migration takes long time, we need to report the progress. We also need to be able to abort migration. The work is mostly done, patches are pending review. We are discussing the possible concerns related to them. E.g. to kill it or to wait until it finishes. If we use block copy `dd` with 32MB chunk size, we have abort points each chunk.
    
    RemoteFS driver refactoring. RemoteFS's hypervisor needs different volume/snapshot format, but nova first-class provides qcow2. Nova-assisted snapshots (when volume is in attached state) currently qcow2 only. It is hard to add support for a new volume type. Spec at https://review.openstack.org/#/c/237094/1/specs/mitaka/remotefs-volume-format-handlers.rst. It is proposed to add volume format base class and volume snapshot format base class and add volume format/snapshot handlers for each type. The other cores seem not very accepted with that. The work remains in question/re-work.
    
    LVM driver, the reference impl, needs love too but it is rarely cared; everyone doing presentation says don't use that. Is LVM still proper as the reference impl? How about Ceph? But ceph uses its own protocol rbd, rather than iscsi. DRBD status and how to support all the features. LVM thin provision should be enabled (rather than thick on default). We can prototypely implement CG in LVM driver for testing purpose. 
    
    Make rally job voting for new patch test? This adds performance verification for Cinder. The concern is John discussed with Rally team and they said Rally is not stable enough. The performance results are not stable enough. We don't want to add another block against patches being merged. http://logs.openstack.org/05/238505/1/check/gate-rally-dsvm-cinder/1c9ae79/. Another concern is that too few people in the community may be aware or interested in the rally part.
    
    Third party CI. We need to revise the status. Which works, which not, and how we want it to grow. Very need to improve devstack and tempest outputs to make finding bug easier. CI Watch http://ci-watch.tintri.com/project?project=cinder. You can find which test is totally useless here because it never succeeds.
    
    Priorities: H/A, rolling upgrade, microversions before M1
    
Other Thinkings

    I found there are too few in Openstack community who are fully aware of container technologies, even if we have Magnum for long time. That's too few people compared with such a disruptive technology.
    
    Both Murano and Magnum provides k8s, but the community looks diverged on them. Mirantis presentations provide both Murano and Magnum but doesn't mention the divergence or integration. "For those who don’t want to use Murano, the OpenStack Magnum project ..." in Mirantis article (https://www.mirantis.com/blog/yes-containers-need-openstack/) seems to have presumed the divergence. The Magnum presentations never mentioned Murano in my impression. So I guess those two are really diverged, but not publicly aware yet. Thanks for my colleague to think of this.