Tasks By Project

Project: Data-Engineering-Kanban

Switch to Active Tasks 18 Phabricator task(s).

Phabricator Link	Wiki Link	Status	Priority	Author	Assignee	Projects	Subtasks	Parent Tasks
T215001	T215001: Revisions missing from mediawiki_revision_create	resolved	High (red)			Patch-For-Review Analytics (Data Quality) Data-Engineering-Kanban (Done) Event-Platform (In Progress) Growth-Team (External) Product-Analytics (Tracking) Growth-Team-Filtering MW-1.37-notes (1.37.0-wmf.5; 2021-05-11) Analytics-Kanban	T280538: Capture rev_is_revert event data in a stream different than mediawiki.revision-create	T120242: Consistent MediaWiki state change events | MediaWiki events as source of truth
T216504	T216504: page-links-change stream is assigning template propagation events to the wrong edits	open	Medium (orange)			Analytics (Incoming) Patch-For-Review Event-Platform (Backlog) The-Wikipedia-Library (Backlog) Data-Engineering-Kanban (Next Up) Data-Engineering (Instrument) Internet-Archive (Links) Platform Team Workboards (Clinic Duty Team) (Later)		T115119: Create a feed or log of changed links on Wikimedia projects
T249755	T249755: Cassandra3 migration for Analytics AQS	open	High (red)			Data-Engineering-Kanban (Q2 Epics) Data-Engineering (Epics) Cassandra (Backlog) Epic Platform Team Workboards (Platform Engineering Reliability) (Watching) Analytics (Analytics-Clusters) (Q2 2021/2022)	T255141: Upgrade the Cassandra AQS cluster to Cassandra 3.11 T280649: Add a spark job loading Cassandra 3 T278699: AQS Cassandra driver needs to be updated T257572: Set up a testing environment for the AQS Cassandra 3 migration T289161: Update cassandra oozie jobs to load cassandra3 using Spark job T278701: Store AQS schema and grants in git T249756: Cassandra3 migration plan proposal T280155: Dual loading from Hive into old and new AQS clusters T290068: Check AQS with cassandra (serving + data) T297460: Send cassandra3 (new hosts) logs to logstash T297803: Switch over the Cassandra AQS cluster to the new hosts T298516: Investigate high levels of garbage collection on new AQS nodes
T255141	T255141: Upgrade the Cassandra AQS cluster to Cassandra 3.11	duplicate	High (red)			Platform Team Workboards (Platform Engineering Reliability) (Backlog) Analytics-Kanban Data-Engineering (Incoming) Data-Engineering-Kanban (Next Up) Analytics (Analytics-Clusters) (Q1 2021/2022)		T249755: Cassandra3 migration for Analytics AQS
T257572	T257572: Set up a testing environment for the AQS Cassandra 3 migration	open	Needs Triage (violet)			Cassandra (Backlog) Patch-For-Review Analytics (Analytics-Clusters) (Q2 2021/2022) Data-Engineering-Kanban (Next Up) Analytics-Kanban Data-Engineering (Serve)		T249755: Cassandra3 migration for Analytics AQS
T282131	T282131: Determine which remaining legacy EventLogging schemas need to be migrated or decommissioned	open	High (red)			Data-Engineering (Ingest) Better Use Of Data (Inbox) Event-Platform (In Progress) Fundraising-Backlog (Blocked or not fr-tech) Data-Engineering-Kanban (In Progress) Product-Analytics (Tracking) Product-Data-Infrastructure (Inbox) Analytics-Kanban Analytics-EventLogging (Backlog)	T283551: Decide whether to keep the ExternalGuidance data stream T290303: Migrate WikibaseTermboxInteraction EventLogging Schema to new EventPlatform thingy	T259163: Migrate legacy metawiki schemas to Event Platform
T285692	T285692: Write a job entirely in Airflow with spark and/or sparkSQL	open	High (red)			Patch-For-Review Airflow (Done) Data-Engineering-Kanban (Done) Data-Engineering (Transform)		T284172: [SPIKE] analytics-airflow jobs development
T288271	T288271: Make it possible to use anaconda + stacked conda envs for Airflow executors	declined	High (red)			Data-Engineering (Transform) Data-Engineering-Kanban (Next Up) Airflow (Next Up)		T282033: Airflow collaborations T294024: [Airflow] Automate sync'ing archiva packages to HDFS
T289161	T289161: Update cassandra oozie jobs to load cassandra3 using Spark job	resolved	High (red)			Data-Engineering-Kanban (Next Up) Epic Cassandra (Backlog) Analytics-Kanban Analytics (Smart Tools for Better Data) Data-Engineering (Incoming)		T249755: Cassandra3 migration for Analytics AQS
T290068	T290068: Check AQS with cassandra (serving + data)	resolved	High (red)			Data-Engineering-Kanban (Next Up) Analytics (Analytics Query Service) Epic Data-Engineering (Serve) Analytics-Kanban	T291470: Repair and reload cassandra2 mediarequest_per_file data table T291469: Repair and reload all cassandra-2 data tables but the 2 big ones T291472: Snapshot and Reload cassandra2 pageview_per_article data table from all 12 instances T291473: Test snapshot-reload from all instances using pageview-top data table	T249755: Cassandra3 migration for Analytics AQS
T290303	T290303: Migrate WikibaseTermboxInteraction EventLogging Schema to new EventPlatform thingy	open	High (red)			Event-Platform (Backlog) Wikidata-Campsite (Incoming) Data-Engineering (Ingest) Data-Engineering-Kanban (In Progress) Analytics (Event Platform) Wikidata (incoming) Wikidata-Termbox (Backlog) wdwb-tech (Active)		T282131: Determine which remaining legacy EventLogging schemas need to be migrated or decommissioned
T291470	T291470: Repair and reload cassandra2 mediarequest_per_file data table	resolved	High (red)			Data-Engineering (Serve) Data-Engineering-Kanban (Done) Analytics-Kanban		T290068: Check AQS with cassandra (serving + data)
T291472	T291472: Snapshot and Reload cassandra2 pageview_per_article data table from all 12 instances	resolved	High (red)			Analytics-Kanban Analytics (Analytics Query Service) Data-Engineering (Serve) Data-Engineering-Kanban (Done)		T290068: Check AQS with cassandra (serving + data) T291473: Test snapshot-reload from all instances using pageview-top data table
T294024	T294024: [Airflow] Automate sync'ing archiva packages to HDFS	open	High (red)			Airflow (Done) Data-Engineering-Kanban (Ready to Deploy) Data-Engineering (Transform)	T288271: Make it possible to use anaconda + stacked conda envs for Airflow executors	T282033: Airflow collaborations
T294026	T294026: [Airflow] Create repository for Airflow DAGs	open	High (red)			Data-Engineering-Kanban (Done) Airflow (Done) Data-Engineering (Transform)		T282033: Airflow collaborations
T297460	T297460: Send cassandra3 (new hosts) logs to logstash	open	High (red)			Data-Engineering-Kanban (Done) Data-Engineering (Ops)		T249755: Cassandra3 migration for Analytics AQS
T297803	T297803: Switch over the Cassandra AQS cluster to the new hosts	open	High (red)			Platform Team Workboards (Platform Engineering Reliability) (Backlog) Analytics (Analytics-Clusters) (Backlog) Epic Cassandra (Backlog) Data-Engineering (Serve) Data-Engineering-Kanban (Paused)		T249755: Cassandra3 migration for Analytics AQS
T298516	T298516: Investigate high levels of garbage collection on new AQS nodes	open	High (red)			Data-Engineering-Kanban (In Progress) Analytics (Analytics-Clusters) (Backlog) Cassandra (Backlog) Epic Data-Engineering (Serve) Platform Team Workboards (Platform Engineering Reliability) (Backlog)		T249755: Cassandra3 migration for Analytics AQS