Monitoring and Measuring

Auditing — SetupAuditTrails for subscribers are aggregated via the API and scanned for unhealthy changes. (TBD)
Defect Analytics — Defect dashboards and reports are available in the business org. (TBD)
Defect KPI — Defect reports and repairs are closely monitored and used as a key performance indicator. (TBD)
Defect Resolution — Crucial defects are quickly repaired and (MP) made available as patch versions to affected subscribers. (TBD)
Defect Timebox — Rather than assign story points, initial work on a defect proceeds within a predefined standard timebox. If the defect cannot be resolved within the timebox, then whether to proceed is reviewed at standup. For example, the timebox may be based on how long it takes to resolve 80% of the defects (TBD)
Error Email Logging — (sf) Application error emails are forwarded to an inbound email service in the business org. (TBD)
Fail Events — (sf) Issues found by failing Apex Unit tests or Selenium tests are posted to cloud-based logging systems, just like production errors and messages. (TBD)
Feature Auditing — (sf) FeatureSetupAuditTrail for subscribers are aggregated as part of Push Metrics. (Needs to be considered in the context of the Feature Management pilot.) (TBD)
Hammer Tests — (sf) Subscriber Apex tests are run periodically to detect any new failures. (TBD)
Heath Predictors — Defect Magnets; Hidden Features; Unilateral Changes (TBD)
Information Radiator — Display key statistics, performance indicators, and inspirational messages in a highly visible location, for the benefit of team members and passers-by. (TBD)
Known Issues — Defects are reported to a public site for awareness and feedback that all subscribers can access. (TBD)
Production Immune System — Usage metrics and logs are continually scanned to detect statistically significant variations in error volume, creating a "production immune system". (TBD)
Push Metrics — Feature usage and adoption is automatically monitored and subscriber orgs push configuration settings, logs, and statistics back to the MP business org. For Salesforce, the metrics include data storage used, installed packages, and Lightning adoption. For Communities, metrics include license counts and Community logins. (TBD)
Remote Event Logging — (TBD)
Root Cause Analysis — To ensure the quality and improvement of our change management process, and to prevent previously discovered problems from recurring, we perform a root cause analysis (RCA), by asking five whys. (TBD)
Score Card — A set of leading and lagging metrics are used to predict and then review the success of a CS project or MP major version and compared with actual results prior to the next event. (TBD)
Trust Dashboard — When reliability issues are detected, the incident is posted to a system health dashboard that all subscribers can access. (TBD)
Usage Metics — For each subscriber, tally record creation volume, page access volume, and page render times, and summarize on a dashboard. (TBD)