Beyond the Screenshot: Why Auditors Don't Trust Platforms & What Quality Really Costs w/ Troy Fine
GRC EngineerShow Notes
Paramify is making FedRAMP (Rev 5 or 20x), GovRAMP & CMMC fun. Get your $750 Gap Assessment at paramify.com/grc---Troy Fine has conducted hundreds of SOC 2 audits over 15 years. In this conversation, he reveals uncomfortable truths about the audit market that most practitioners won't discuss openly.His most explosive admission: "Nobody can measure audit quality." Not TPRM teams. Not buyers. Not even auditors themselves. You're not paying for quality - you're paying for brand recognition.We cover:**The Evidence Trust Problem**Why auditors trust screenshots but not platform automation, the middleware accountability gap that makes audit firms uncomfortable, and what professional liability concerns reveal about legal defensibility versus technical capability.**Quality vs Brand Reality**Troy's admission that even premium audit firms don't provide measurably better quality, why personal brand premium pricing works at small scale but doesn't solve systematic problems, and how the audit market operates on reputation signalling rather than measurable outcomes.**Platform Evidence & Professional Liability**The risk-based framework Troy actually uses: accepting platform evidence for low-risk controls whilst validating source systems for infrastructure, what would make platforms auditor-trustworthy (cryptographic evidence chains, auditor-controlled queries, platform certification), and why the courtroom scenario keeps auditors sceptical of automation.**SOC 2 Market Commoditisation**The feedback loop problem driving quality degradation, why "no report is better than bad report" reveals systematic market failure, the two-tier market emerging (premium craftsmanship versus commoditised checkbox exercises), and how price compression without quality metrics creates race-to-bottom dynamics.**The SOC 2 Lite Proposal**Troy's vision for formal tiered assurance with 20 prescriptive controls for smaller companies, why this would fail in practice (TPRM teams defaulting to "Full," gaming qualification criteria, arbitrary thresholds), and what transparency about validation depth would actually provide instead.**AI in Audit Practice**Where Troy embraces AI (evidence evaluation, pattern detection, documentation efficiency) versus where human judgement remains essential (risk assessment, control design evaluation, professional scepticism), and why accountability architecture matters more than tool ownership.**What Would Actually Fix This**Moving from point-in-time audits to continuous assurance, building cryptographic evidence chains for provenance verification, auditing platform methodology once instead of each deployment, and why engineering discipline with measurable quality metrics could replace subjective professional judgement.
Connect with Troy:LinkedIn: https://www.linkedin.com/in/troyjfine/Fine Assurance: fineassurance.com**About The GRC Engineer:**The GRC Engineer explores how engineering principles are transforming governance, risk, and compliance. Hosted by Ayoub Fandi, each episode features practitioners, leaders, and innovators building the future of GRC through automation, code, and systems thinking.🌐 Visit: grcengineer.com💼 Connect: linkedin.com/in/ayoubfandi📧 Newsletter: grcengineer.com/subscribeSubscribe for deep-dives into GRC automation, compliance as code, risk engineering, and the intersection of security, compliance, and software development.#GRCEngineering #SOC2 #Audit #Compliance #TroyFine #CyberSecurity #RiskManagement #Automation #SecurityCompliance #AuditQuality
More from GRC Engineer
Every compliance framework you know was built for deterministic systems. AI agents are not deterministic. That's why AIUC-1 was born.In this episode, I sit down with Danny from Schellman and Rajiv Dattani, co-founder of AIUC, to break down the first compliance framework purpose-built for AI agents. We cover the six pillars (data & privacy, security, safety, reliability, accountability, societal risks), how the technical testing works with thousands of adversarial simulations, and why tying it to insurance through Lloyd's of London changes the incentive structure for the entire audit market.Key takeaways:- AIUC-1 bridges the gap between governance frameworks (ISO 42001) and technical testing- You can't commoditize it: you either pass thousands of adversarial scans or you don't- Quarterly updates keep the framework current with the fast-moving AI threat landscape- Insurance-backed testing means the tester has skin in the game- The 100-page audit report replaces ~70% of your vendor questionnaire back-and-forth- Three evidence types: adversarial testing, technical controls (code review), and policies- Internal agents emphasize security/data privacy; external agents emphasize safety---CHAPTERS:00:00 Introduction & Guest Intros01:55 Why a New Framework for AI Agents?04:06 Why Schellman Partnered with AIUC08:06 Quarterly Updates & The Consortium Model09:29 The Five Principles Driving AIUC-111:36 The Six Pillars: Data, Security, Safety, Reliability, Accountability, Societal12:38 What You Get: 100-Page Report + Certification14:22 How Testing Works: Thousands of Adversarial Simulations16:12 Testing Stochastic Systems: The Entropy Problem17:05 The Insurance Innovation: From Benjamin Franklin to AI Safety20:39 Lloyd's of London & Synthetic Loss Data21:36 Aligning Incentives Through Insurance23:56 How Enterprises Use AIUC-1: Buyers vs Builders26:06 Deterministic vs Stochastic: The Compliance Challenge29:05 SOC 2 vs ISO 42001 vs AIUC-1 Positioning30:51 Threat Modeling Meets Compliance35:10 Traditional Security Controls & AI-Specific Risks38:11 GRC Engineering & Agentic Auditing42:18 The Hardest Challenge: Articulating Technical Testing44:46 Closing Thoughts------CONNECT WITH GRC ENGINEER:Newsletter: https://grcengineer.com/subscribeLinkedIn: https://linkedin.com/in/ayoubfandiWebsite: https://grcengineer.comCONNECT WITH DANNY:LinkedIn: https://www.linkedin.com/in/danny-manimbo-2b199718/CONNECT WITH RAJIV DATTANI:LinkedIn: https://linkedin.com/in/rajivdattani---#GRC #cybersecurity #compliance #AIagents #AIUC1 #ISO42001 #GRCEngineering
Paramify is making FedRAMP (Rev 5 or 20x), GovRAMP & CMMC fun. Get your $750 Gap Assessment at paramify.com/grc
Paramify is making FedRAMP (Rev 5 or 20x), GovRAMP & CMMC fun. Get your $750 Gap Assessment at paramify.com/grc. To get access to the deep-dive transcript, subscribe to the GRC Engineer newsletter: grcengineer.com/subscribe Wrong ink colours. $300,000 authorizations. Congressional investigations within the first month. How do you fix federal compliance from the inside?In this episode, Pete Waterman, Director of FedRAMP, shares how he's applying 20+ years of engineering experience to rebuild federal authorization from first principles. What started with "violent hatred" of the programme has become one of the most significant transformations in government compliance.Pete's approach is radically different: treat policy like code, make the secure thing the easy thing, and let engineers lead whilst compliance follows. The results speak for themselves. Key Topics Discussed: The Problem State How FedRAMP became a programme where perfection was fetishised beyond security, packages were rejected for cosmetic issues, and $300k costs prevented small teams from using modern tools FedRAMP 20X Architecture The dual-path strategy: improving Rev5 whilst building something entirely new with Key Security Indicators, machine-readable evidence, and persistent validation Risk-Based Authorization Why "my job is to make the government take more risks" - moving from bar-based to spectrum-based assessment where agencies choose based on their risk tolerance Engineering-First Requirements How KSIs like "prevent unauthorized access" replace "do these 18 specific things" and why cloud-native thinking changes everything Radical Transparency Doctrine Why posting roadmap updates every two weeks on GitHub creates trust and how "pre-decisional" anxiety is outdated thinking About the Guest: Pete Waterman is Director of FedRAMP, bringing over 20 years of engineering leadership experience to federal compliance. Previously worked with US Digital Service as a cloud expert, the Technology Modernization Fund coaching agencies on modernization, and ran engineering at an AI company. He took over FedRAMP in August 2023 with a mandate to transform the programme from an engineering-first perspective. Connect with Pete: Pete Waterman: https://www.linkedin.com/in/petewaterman/ About The GRC Engineer: The GRC Engineer explores how engineering principles are transforming governance, risk, and compliance. Hosted by Ayoub Fandi, each episode features practitioners, leaders, and innovators who are building the future of GRC through automation, code, and systems thinking. Subscribe for episodes and entries featuring deep-dives into GRC automation, compliance as code, risk engineering, and the intersection of security, compliance, and software development. 🌐 Visit: grcengineer.com 💼 Connect: linkedin.com/in/ayoubfandi 📧 Newsletter: grcengineer.com/subscribe #GRCEngineering #FedRAMP #Compliance #Automation #CyberSecurity #RiskManagement #DevSecOps #CloudSecurity
To get access to the deep-dive transcript, subscribe to the GRC Engineer newsletter: grcengineer.com/subscribe How do you build a modern GRC programme when you inherit processes designed for a team three times your size, in an organisation where "compliance frameworks were owning us instead of us owning them"? In this episode, Emre Ugurlu and Chad Fryer from Docker share their journey transforming compliance, risk, and customer trust functions over the past six months through relentless automation, AI-assisted development, and a ruthless focus on user experience. Emre previously spent 3.5 years at Plaid working on GRC engineering principles, whilst Chad brings a UX focus with a strong engineering background. Together with a small team at Docker, they're proving that you don't need a massive GRC organisation to deliver enterprise-grade compliance at speed. Build vs Buy Philosophy Why Docker defaults to internal development and how they rebuilt their entire security training platform in a couple of weeks, achieving 100% completion rates through gamification and automation. Zero-to-One Playbook The first weeks: deep gap analysis, stress-testing controls, collaborative stack-ranking across teams, and building communication channels before building solutions. Self-Managing Team Model Three engineers, one analyst, no dedicated GRC manager. How autonomy and trust from leadership enables speed and innovation. Continuous Compliance at Scale Moving towards full automation across SOC 2 and ISO 27001, including custom API development with AWS Lambda and EventBridge. AI as Teammate Claude as "the sixth member" of the team, the discipline required to use AI effectively, and why pre-AI coding experience makes you 10x better at leveraging it. User Experience in GRC Why if nobody uses your solution, it doesn't matter how good it is. Building for adoption, not perfection. TPRM Transformation "We promised Steven we would automate the crap out of it" - plans for comprehensive third-party risk management automation. Cost Model Innovation How Docker's GRC team is becoming a revenue-generating function by saving costs and offering solutions to other internal teams. Essential Skills What aspiring GRC engineers actually need: API documentation reading, embracing failure, proper documentation, and understanding code across multiple languages. 12-Month Vision Open source tool releases, containerised solutions for the community, and the goal to "transform GRC into something no one's ever seen." Open source cybersecurity training already available: https://emreugurlu.github.io/open-security-training/ Quotes: "Instead of bending over backwards, we're supposed to make it fit the organisation. Docker is really unique in the way it operates, and we have to adjust compliance accordingly." - Emre "If we build the most cool thing on the planet, but nobody uses it, it doesn't matter. Everything I do, I think of user experience during the process." - Chad "Six times out of ten, I have to go correct Claude. The ability to read through code and read through flawed logic never disappears." - Emre "With the tools we have today, there's no excuse why anybody can't build things themselves." - Emre "We're going to be a revenue generating team." - Chad About The GRC Engineer: The GRC Engineer explores how engineering principles are transforming governance, risk, and compliance. Hosted by Ayoub Fandi, each episode features practitioners, leaders, and innovators who are building the future of GRC through automation, code, and systems thinking. Subscribe for episodes and entries featuring deep-dives into GRC automation, compliance as code, risk engineering, and the intersection of security, compliance, and software development. 🌐 Visit: grcengineer.com 💼 Connect: linkedin.com/in/ayoubfandi 📧 Newsletter: grcengineer.com/subscribe
Check out grcengineer.com to learn more!SummaryIn this engaging conversation, Ayoub Fandi and Varun Gurnaney explore the evolving landscape of Governance, Risk, and Compliance (GRC) engineering. Varun shares his unique journey from cybersecurity to GRC, emphasizing the importance of automation and collaboration between engineering and compliance teams. They discuss the challenges faced in GRC, the philosophical aspects of risk management, and the future of compliance in a rapidly changing technological environment. The dialogue highlights the need for a more integrated approach to security and compliance, advocating for a shift towards real-time assessments and a deeper understanding of the technical landscape.Sound Bites"Screenshots are cool again.""Compliance should be free.""Don't get hacked is what I care about."TakeawaysVarun's journey into GRC began with a cybersecurity role at EY.The importance of automation in GRC processes is crucial for efficiency.Cultural differences in compliance approaches between small and large companies.GRC engineering is often misunderstood and underappreciated in larger organizations.The need for collaboration between GRC and engineering teams is essential for success.Risk management should be tied to real business impacts rather than just compliance checkboxes.The future of compliance may involve more automated and real-time assessments.Tools used in security can significantly enhance GRC efforts.Understanding the technical landscape is vital for effective GRC practices.The conversation highlights the philosophical aspects of compliance and risk management.Chapters00:00 Introduction and Guest Background02:42 Varun's Journey into GRC Engineering06:32 Comparing GRC in Different Company Sizes11:56 The Role of Automation in GRC17:34 Challenges in GRC Engineering23:26 The Future of Compliance and Risk Management29:03 The Importance of Collaboration in Security34:47 The Philosophy of Risk and Compliance40:33 The Role of Tools in GRC46:21 Final Thoughts on GRC and Future Directions