Share this Job

Senior Testing Analyst

Date:  Feb 16, 2021

Broomfield, CO, US

Company:  Vail Resorts
Shift Type:  Year Round

Reach Your Peak at Vail Resorts.  As a community of adventurers and discoverers, Vail Resorts delivers an experience of a lifetime to our guests and our employees. Our team is made whole by the brave, passionate individuals who ambitiously push boundaries and challenge the status quo. Whether you’re looking for seasonal work or the career of a lifetime, join us today to reach your peak. 


Job Summary

The primary purpose of the Senior Testing Analyst position is to provide recommendations for test and personalization improvements, monitor performance of live A|B testing across primary marketing audiences and channels, identify reporting opportunities for test performance reads, and partner with other Analytics and Marketing teams. You will partner with Digital Experience, Media, Relationship Marketing, Analytics, and IT teams to define test hypotheses, identify success/failure criteria, and select representative audience samples. You will also coordinate with the IT team to identify opportunities for improved efficiency and to ensure ongoing reporting is created in the most standardized manner possible. Your involvement in Marketing includes: being the liaison between Analytics / Channel Marketing for all test setup, ensuring recurring analysis and reporting processes are conducted correctly and in a timely manner, analyzing test results, providing recommendations, troubleshooting, and producing detailed ad-hoc analysis as required by the business leaders. You will be responsible for helping senior leaders understand: A|B & multivariate testing effectiveness, confidence thresholds, and using Test & Learn software to forecast roll-out of winning test experiences.


Job Responsibilities

  • Partner with Marketing & Channel teams to understand go-to-marketing strategies, business objectives, and key questions. Leverage prior test performance and new insights to provide testing recommendations (defining the hypothesis, reporting KPIs, audience segments, traffic splits, etc.)
  • Understand Test & Learn tooling and setup process, define success/failure criteria across key KPI’s, check & mitigate against sample bias, advise & measure against confidence levels, and propose audience splits
  • Partner with Analytics, Channel, and IT teams to provide ongoing test reads, reporting, and actionable insights. Understand and be able to interpret lift impacts and confidence levels using business acumen. Advise high-impact audiences for roll-out of winning experiences. Forecast and measure testing lift against baseline business results
  • Partner with Test & Learn teammates to align processes, facilitate trainings, and document best practices that ensure efficient and consistent analysis across the team.  Standardize methodology for addressing common questions and ensure that consistent approaches be utilized across the enterprise.  Proactively develop new analytic approaches and partner with other analytical groups to develop tools that support marketing tactics at every level
  • Create and distribute recurring test read-outs & reports to track existing test performance and provide insights that inform new testing opportunities for the enterprise
  • Analyze and organize the testing data available for business use.  Inspect, clean, understand, and transform data as required for analysis purpose. Keep the big picture of business goals in mind while thinking of new ways to approach analysis and enterprise-level testing insights
  • Support complex ad-hoc pre, during, and post analysis of tests as required by leadership and partner teams
  • Leverage knowledge of test setup, site & campaign implementation, and key business KPIs to partner with IT teams to quickly troubleshoot issues as they arise
  • Populate and maintain accessible knowledge base of past testing to enable trended impact of Test & Learn impact to business results, provide guidance on methodology & assumptions used in analysis, and to enforce integrity of testing process and use of best practices across the enterprise. Leverage historical learnings to make recommendations on strategy to marketing leadership & lead culture changes to enable an agile Test & Learn environment


Job Requirements


  • Bachelor’s degree in related field (Business, Technology, Economics, Marketing, Statistics, Computer Science, Data Science, or Mathematics)
  • 3-5 years prior marketing analytics, A|B testing, statistics, digital analytics, or other related experience
  • Experience evaluating website or marketing channel performance, including analysis of A|B experiments and evaluation of results against randomized control groups
  • Statistics background including but not limited to A|B Testing, Multivariate Testing, Regression Analysis, and Hold-out Experiments
  • Experience with 1 or more analytics tools/software specific to testing analysis (Adobe Analytics / Adobe Target / Optimizely / Google Analytics / APT / other CRO platforms). RStudio, Python, Alteryx, Tableau, experience a plus


  • High level of comfort with Digital Analytics & Marketing Technology
  • Demonstrated success working with very large data sets, developing tools for streamlined analysis, and creating reports that succinctly summarize complex concepts
  • Comfortable running statistical analysis and manipulating data in Alteryx, RStudio, Python, or other open source analytics software
  • Strong organizational and project management skills
  • Comfortable presenting to and communicating with senior leadership



The budgeted range starts at $71,500-$88,200 + annual bonus.  Actual pay will be adjusted based on experience.

The perks include a free ski pass, and a set of benefits including... 

  • Medical, Dental, Vision insurance, and a 401(k) retirement plan  
  • Hourly employees are generally eligible for accrued Paid Time Off (PTO), Extra Time Off (ETO) and Sick Time.  Salaried employees are generally eligible for Flexible Time Off (FTO) 
  • Paid Parental Leave for eligible mothers and fathers  
  • Healthcare & Dependent Care Flexible Spending Accounts  
  • Life, AD&D, and disability insurance 


We offer a variety of career opportunities at our world-class resorts and corporate headquarters near Boulder, Colorado in fields like Finance & Accounting, Human Resources, Information Technology, Legal, Public Affairs & Sustainability, Marketing, Sales & Communications and more. Our corporate team shares both a passion for the outdoors and a drive to re-imagine the mountain resort experience around the world. Learn more at www.vailresortscareers.com

Vail Resorts is an equal opportunity employer. Qualified applicants will receive consideration for employment without regard to race, color, religion, sex, national origin, sexual orientation, gender identity, disability, protected veteran status or any other status protected by applicable law.

Requisition ID 286952

Nearest Major Market: Denver