These tools are all oriented towards migration of external data sources to phab.

This repo will eventually be used for tooling surrounding development in phabricator and related tools.

For the moment it is being used to house migration scripts and associated logic.

Everything here is a work in progress.

Preflight check for migrations:

set up security group
set up security herald rule
-- make sure ext ref key is editable --
-- make sure icons are excluded
make sure bot is in security group
ensure file size (mysql / php)
bot creation
set up hosts file entry for db on main (dns is slow)
create and modify /etc/phabtools.conf (start from phabtools.conf.example)

• bot user must be a member of importbots group
• /etc/phabtools.conf must exist

OPTIONAL:

• loading a list of "fake" people for testing can be done for RT why haveing a dict of users in rtppl.py via test/loadrtppl.py ppl = {"test": "test@wikimedia.org"}

OPERATION

See bugrun.sh

bugzilla_fetch.py reads Bugzilla, populates a local database next to Phabricator. A lot of the data is stored as JSON in its fields.

Then bugzilla_create.py creates tasks. bugzilla_update_tasks.py updates tasks

bugzilla_populate_user_relations_comments_table.py and bugzilla_populate_user_relations_table.py get the comment transaction ID. If the user exists in Phabricator they can later associate the right user with it.

bugzilla_update_user_comments.py updates comments

This repo makes use of the following packages:

python-phabricator (sudo apt-get install python-phabricator

or git clone https://github.com/disqus/python-phabricator)

python-rtkit python-mysqldb (sudo apt-get install python-mysqldb)

Notes on DB Schema for migration:

create database bugzilla_migration; create database rt_migration; create database fab_migration;

use bugzilla_migration; CREATE TABLE bugzilla_meta (

 id INT,
 priority INT,
 header TEXT(10000),
 comments TEXT(100000),
 xcomments TEXT(100000),
 created int(12) NOT NULL,
 modified int(12) NOT NULL
);

use rt_migration; CREATE TABLE rt_meta (

 id INT,
 priority INT,
 header TEXT(10000),
 comments TEXT(100000),
 xcomments TEXT(100000),
 created int(12) NOT NULL,
 modified int(12) NOT NULL
);

use fab_meta; CREATE TABLE fab_meta (

 id INT,
 priority INT,
 header TEXT(10000),
 comments TEXT(100000),
 created int(12) NOT NULL,
 modified int(12) NOT NULL
);

CREATE TABLE user_relations (

id INT NOT NULL AUTO_INCREMENT PRIMARY KEY,
priority INT,
user TEXT(1000),
assigned TEXT(10000),
cc TEXT(10000),
author TEXT(10000),
created int(12) NOT NULL,
modified int(12) NOT NULL

);

CREATE TABLE user_relations_comments (

id INT NOT NULL AUTO_INCREMENT PRIMARY KEY,
priority INT,
user TEXT(1000),
issues TEXT(10000),
created int(12) NOT NULL,
modified int(12) NOT NULL

);

CREATE TABLE user_relations_jobs (

                                  id INT NOT NULL AUTO_INCREMENT PRIMARY KEY,
			          pid INT,
                                  source TEXT(30),
			          start INT(12),
			          finish INT(12),
			          status INT(12),
			          start_epoch INT(12),
			          finish_epoch INT(12),
			          user_count INT,
			          issue_count INT,
			          completed INT,
			          failed INT,
			          modified INT(12)

);

create table task_relations (id INT, priority INT, blocks TEXT(1000), modified INT(12));

Issue examples:

Consistent Attachments:

https://old-bugzilla.wikimedia.org/show_bug.cgi?id=72256

BZ lots of attachments:

https://bugzilla.wikimedia.org/show_bug.cgi?id=22881 has 11 https://bugzilla.wikimedia.org/show_bug.cgi?id=6068 has 14 https://bugzilla.wikimedia.org/show_bug.cgi?id=55332 has 7, patches and html https://bugzilla.wikimedia.org/show_bug.cgi?id=19991 has 5

BZ attachment since deleted:

https://bugzilla.wikimedia.org/show_bug.cgi?id=2 has 1

RT ISSUES:

https://rt.wikimedia.org/Ticket/Display.html?id=8175

http://requesttracker.wikia.com/wiki/REST