Share this Job

Senior Analyst, Channel Testing

Date: Jul 2, 2019

Location: Broomfield, CO, US

Company: Vail Resorts

Reach Your Peak at Vail Resorts. You're someone who pushes boundaries and challenges the status quo. You're brave, ambitious and passionate in everything you do. And we want you on our team. Pursue your fullest potential and never settle in the quest to deliver extraordinary guest service. Join one of the world's most innovative companies and re-imagine a mountain resort experience with us. Welcome to Vail Resorts. Reach Your Peak.

Welcome to Vail Resorts. Reach Your Peak.

Summary of Job:


Department Purpose


The Channel Analytics team is responsible for: using a data driven approach to inform marketing decisions, ensuring testing excellence, and maximizing channel effectiveness across the organization. Channel Insights assesses the performance of RM, Media, Digital, Social, Direct Mail and AI marketing efforts in order to provide actionable insights and reporting that inform marketing decisions across both channel and business teams. Test & Learn leverages data and prior test performance to partner with Analytics and Marketing teams to effectively test and measure impact of personalization and optimization improvements across RM, Media, Digital, Social, Direct Mail, and AI-driven marketing channels.


Positions Purpose


The primary purpose of the Sr. Testing Excellence analyst position is to provide recommendations for test and personalization improvements, monitor performance of live A|B testing across primary marketing audiences and channels, identify reporting opportunities for test performance reads, and partner with other Analytics and Marketing teams to design, develop, and implement new testing ideas based on key insights gleaned from the data. You will partner with Digital Experience, Media, Relationship Marketing, Analytics, and IT teams to define test hypotheses, identify success/failure criteria, and select representative audience samples. You will coordinate with the IT team to identify opportunities for improved efficiency and to ensure ongoing reporting is created in the most standardized manner possible. Your involvement in Marketing includes: being the liaison between Analytics / Channel Marketing for all test setup, ensuring recurring analysis and reporting processes are conducted correctly and in a timely manner, analyzing test results, providing recommendations, troubleshooting, and producing detailed ad-hoc analysis as required by the business leaders. You will be responsible for helping senior leaders understand: A|B & multivariate testing effectiveness, test impacts against baseline business performance, confidence thresholds, and using Test & Learn software to forecast roll-out of winning test experiences. Finally, you will be expected to partner closely with other Test & Learn team members to align on process, enforce testing best practices, create documentation and SOP’s, and to help guide the enterprise towards an agile Test & Learn culture.


Job Responsibilities:


  • [Strategy] Partner with Marketing & Channel teams to understand go-to-marketing strategies, business objectives, and key questions. Leverage prior test performance and new insights to provide testing recommendations (defining the hypothesis, reporting KPIs, audience segments, traffic splits, etc.).
  • [Testing Design] Understand Test & Learn tooling and setup process, define success/failure criteria across key KPI’s, check & mitigate against sample bias, advise & measure against confidence levels, and propose audience splits.
  • [Interpretation & Insights] Partner with Analytics, Channel, and IT teams to provide ongoing test reads, reporting, and actionable insights. Understand and be able to interpret lift impacts and confidence levels using business acumen. Advise high-impact audiences for roll-out of winning experiences. Forecast and measure testing lift against baseline business results.
  • [Standardization & Efficiency] Partner with Test & Learn teammates to align processes, facilitate trainings, and document best practices that ensure efficient and consistent analysis across the team.  Standardize methodology for addressing common questions and ensure that consistent approaches be utilized across the enterprise.  Proactively develop new analytic approaches and partner with other analytical groups to develop tools that support marketing tactics at every level.
  • [Reporting] Create and distribute recurring test read-outs & reports to track existing test performance and provide insights that inform new testing opportunities for the enterprise.
  • [Data Analysis] Analyze and organize the testing data available for business use.  Inspect, clean, understand, and transform data as required for analysis purpose. Keep the big picture of business goals in mind while thinking of new ways to approach analysis and enterprise-level testing insights.
  • [Ad-hoc] Support complex ad-hoc pre, during, and post analysis of tests as required by leadership and partner teams.
  • [IT Troubleshooting] Leverage knowledge of test setup, site & campaign implementation, and key business KPIs to partner with IT teams to quickly troubleshoot issues as they arise.
  • [Testing Culture & Knowledge Base] Populate and maintain accessible knowledge base of past testing to enable trended impact of Test & Learn impact to business results, provide guidance on methodology & assumptions used in analysis, and to enforce integrity of testing process and use of best practices across the enterprise. Leverage historical learnings to make recommendations on strategy to marketing leadership & lead culture changes to enable an agile Test & Learn environment.


Job Requirements:


  • Bachelor’s degree in related field (Business, Technology, Economics, Marketing, Statistics, Computer Science, Data Science, or Mathematics) required
  • Master’s degree or PhD in a quantitative / qualitative field preferred, but not required
  • 3-5 years prior marketing analytics, A|B testing, statistics, digital analytics, or other related experience
  • Experience evaluating website or marketing channel performance, including analysis of A|B experiments and evaluation of results against randomized control groups
  • Statistics background including but not limited to A|B Testing, Multivariate Testing, Regression Analysis, and Hold-out Experiments required
  • Demonstrated success working with very large data sets, developing tools for streamlined analysis, and creating reports that succinctly summarize complex concepts
  • Comfortable running statistical analysis and manipulating data in Alteryx, RStudio, Python, or other open source analytics software
  • Demonstrated strength in written and verbal communication, adjusting message to audience.  Excellent interpersonal and relationship management skills, with the ability to communicate effectively at all levels of the organization
  • Strong organizational and project management skills are essential. Must be able handle multiple projects simultaneously and coordinate across multiple functions within an organization
  • Highly collaborative approach and open communication style
  • Comfortable presenting to and communicating with senior leadership
  • Hospitality or travel industry experience helpful but not required
  • Experience with 1 or more analytics tools/software specific to testing analysis (Adobe Analytics / Adobe Target / Optimizely / Google Analytics / APT / other CRO platforms) required. RStudio, Python, Alteryx, Tableau, experience a plus.
  • High level of comfort with Digital Analytics & Marketing Technology
  • High degree of proficiency MS Office Suite, Outlook & Internet applications
  • Knowledge of brand personalities, key product offerings, & resort locations
  • Minimal travel to resort locations and also to partner/vendor locations

We offer a variety of career opportunities at our world-class resorts and corporate headquarters near Boulder, Colorado in fields like Finance & Accounting, Human Resources, Information Technology, Legal, Public Affairs & Sustainability, Marketing, Sales & Communications and more. Our corporate team shares both a passion for the outdoors and a drive to re-imagine the mountain resort experience around the world. Learn more at
Vail Resorts is an equal opportunity employer. Qualified applicants will receive consideration for employment without regard to race, color, religion, sex, national origin, sexual orientation, gender identity, disability, protected veteran status or any other status protected by applicable law.


Requisition ID 168528

Nearest Major Market: Denver