Skip to content

feat(home): cookieless homepage description A/B test (server-side assignment) - #592

Open
mohanadft wants to merge 12 commits into
mainfrom
feat/homepage-cta-experiment
Open

mohanadft wants to merge 12 commits into
mainfrom
feat/homepage-cta-experiment

Conversation

@mohanadft

@mohanadft mohanadft commented Oct 19, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • A/B tests the homepage hero description, with no cookie banner needed:
    • A: the current copy ("Tech for Palestine is a 501(c)(3) nonprofit…" plus the "We support anyone with a meaningful idea…" paragraph).
    • B: the new member-driven copy (four paragraphs, ending with "Our Vision").
  • The variant is picked per request in index.astro (Math.random() < 0.5, src/utils/homepageExperiment.ts) and rendered straight into the HTML. No flicker, no third-party fetch for the experiment, and no JS needed to see either version.
  • Nothing is stored (no cookies, localStorage or sessionStorage). A visitor can get a different variant on each page view, so this only measures outcomes within a single page view.
  • Plausible events, both with experiment: homepage-description and variant: A|B props:
    • Homepage Description Shown: once per page view.
    • Homepage Description CTA Click: at most once per page view, when either hero button ("Become a Member" or "Apply to the Incubator") is clicked, with cta: membership or cta: incubator. Membership clicks are the primary metric; incubator clicks show whether B pulls people away from the incubator.
  • / now sends Cache-Control: no-store, otherwise everyone in the 10-minute cache window would get the same variant.
  • GrowthBook is not used for the test (its stats need a data source, and Plausible isn't one). It stays for feature flags such as the newsletter popup. The homepage-cta-text flag in GrowthBook is unused and can be archived.
  • Layout prerenders / from every page, and prerendered pages run their scripts. The exposure event therefore waits for prerenderingchange, so only page views the visitor actually sees are counted.
  • Middle-clicks (open in new tab) on the hero cards count as clicks.
  • The hero CTAs from feat(volunteer): point volunteers to membership and move page onto live design system #594 become two stacked buttons with the sentence inside and the action in bold: "Join the community, build projects, and start your own through T4P: Become a Member" (primary) and "If you run your own pro-Palestine project, and want help scaling: Apply to the Incubator" (outlined).

Before going live

  • Agree the minimum lift worth detecting, a fixed run length (at least 4 to 5 whole weeks at ~120 homepage views/day) and the decision rule
  • Check the A/B split of shown events looks roughly 50/50 before reading any result
  • Remove Cache-Control: no-store from / when the test ends
  • Add Homepage Description Shown and Homepage Description CTA Click as custom event goals in Plausible
  • Make sure experiment, variant and cta are registered as custom properties in Plausible
  • Optionally delete the old Experiment Exposure / CTA Click goals (test data only)

Reading results

Filter on experiment = homepage-description, break down by variant, and compare click events / shown events per variant (not visitors: one visitor can see both variants).

Test plan

  • pnpm lint && pnpm check && pnpm test
  • Preview: refresh repeatedly, roughly half show each description, no text swap after load
  • Variant B on a laptop-sized viewport: the hero buttons are still visible without scrolling (otherwise B is penalised by layout, not copy)
  • Network: one /api/pipe shown event, at most one click event with the right cta, all with experiment and variant
  • Application tab: no cookies, localStorage or sessionStorage entries from this change
  • Response headers on / show Cache-Control: no-store and the variant differs across requests (not cached at the edge)
  • Chrome: open /about, don't visit /, and confirm no shown event is sent (prerender gating)
  • Preview hostname is not registered on the production Plausible site
  • Check TTFB/LCP on / now that it isn't cacheable

Also in this PR: volunteer form links → membership

  • On /volunteer, every open role's button is now "Become a member" → /membership (project roles used to link to the Formbricks volunteer survey).
  • The open-roles error and empty-state fallbacks link to /membership instead of /volunteer-form.
  • Removed the unused Formbricks survey URL, /volunteer-form constant and role-link builder, plus their tests. No Formbricks URL remains in the repo.
  • Note: the role ID/title/project that the survey captured is no longer recorded.
  • Outside the repo: the /volunteer-form redirect (Cloudflare dashboard) still points at Formbricks; update it to /membership

Hero CTA island reads the homepage-cta-text flag (50% rollout plus a control
force rule, preview only). Assignment uses an in-memory crypto.randomUUID id
with the SDK payload cache disabled, so nothing is written to cookies or
storage. Exposure and CTA click events go to Plausible once per page view with
experiment and variant props.
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Oct 19, 2026 •

Copy link
Copy Markdown

Deploying website with  Cloudflare Pages  Cloudflare Pages

Latest commit: 45aa6bb
Status: ✅  Deploy successful!
Preview URL: https://2080ffae.website-aun.pages.dev
Branch Preview URL: https://feat-homepage-cta-experiment.website-aun.pages.dev

View logs

@mohanadft mohanadft changed the title feat(home): cookieless homepage CTA text test via GrowthBook feat(home): cookieless homepage CTA text test (server-side assignment) Oct 19, 2026
Replace the CTA text test with an A/B test of the hero description.
The control keeps the current two paragraphs; the treatment uses the
new member-driven copy. Assignment stays per request on the server and
nothing is stored. The CTA reverts to "Apply to the Incubator" and its
clicks remain the conversion event.
@mohanadft mohanadft changed the title feat(home): cookieless homepage CTA text test (server-side assignment) feat(home): cookieless homepage description A/B test (server-side assignment) Oct 25, 2026
Events become "Homepage Description Shown" / "Homepage Description CTA
Click" and variants become A / B.
Resolve the hero conflict with #594: keep its two hero cards and track
clicks on both, tagged cta=membership / cta=incubator.
Layout prerenders / from every page and prerendered pages run their
scripts, so the exposure event now waits for prerenderingchange.
Middle-clicks on the hero cards now count as clicks. Tracking takes an
injectable tracker and document so it is unit tested.

Drop the GrowthBook localStorage polyfill: the test no longer uses
GrowthBook, the homepage still writes localStorage elsewhere, and the
polyfill cost a CDN fetch on every page load.
Replace the two hero cards with a primary "Become a Member" pill and an
outlined "Apply to the Incubator" pill. Click tracking is unchanged.
Open-role buttons and the open-roles error/empty fallbacks linked to the
Formbricks volunteer survey. They now all go to /membership. Remove the
unused survey URL, /volunteer-form constant and role link builder.
The hero buttons now read as one line of text with the action in bold:
"Join the community, build projects, and start your own through T4P:
Become a Member" and "If you run your own pro-Palestine project, and
want help scaling: Apply to the Incubator". Click tracking is unchanged.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant