Importing Ascher AS001-AS009

1. Importing Marcia Ascher’s AS001-AS009

Marcia and Robert Ascher described AS001 through AS009 in the journal article Numbers and Relations from Ancient Andean Quipus. Using the source from the original article I have imported their 9 khipus into the Khipu Field Guide (KFG). A few assumptions were made:

  • When a cord was broken, or a knot unreadable, Marcia Ascher always suffixed the value with a question mark. I’ve assumed her best guess was the real number.
  • When she describes cords as made of mixed colors I am assuming those are “mottled” cords, and not barberpole cords.
  • The descriptions lack attachments, spin/ply information, etc., so I default those to unknown.
  • I am trying to match Brezine colors to Ascher colors. I probably did a pretty bad job.

The data was scanned in, using OCR techniques from the original article, and then cross-checked and fixed for each of the khipus, and then stored in Excel files, with four sheets per file:

  • Sheet 1 lists Khipu information (museum, and general notes)
  • Sheet 2 lists primary cord length/color and cord cluster information
  • Sheet 3 lists individual cord information including knots, color, length, and value
  • Sheet 4 lists (some) Ascher colors for that khipu, that need translation into current-day Ascher colors.

2. Data Integrity Checking

Because the original data was scanned in, and because there may still be numerous errors from the OCR process (even though I hand-checked everything), I want to do a data-integrity check for format integrity for each of the 9 khipus.

The approach is to read in each key cell-string, then convert it into data, and then print the data in the same format as the input string. If the input-string matches the output-string, then I will assume the data is properly formatted.

This integrity check resulted in at least fifty small errors being found. I’m now of the opinion that he most productive investment in yak-shaving is data integrity checking.

3. Converting To KFG Excel Spreadsheets

Conversion from Ascher files to the Khipu Field Guide Excel Format was by no means straightforward and took at least one month. Most critical was getting cords and cord clusters correct for pendants, then getting cord colors to line up, and then finally getting knots correct. Along the way, an intermediate format is created, shown below. Finally, the Excel Spreadsheets are written, by first creating complete SQL tables, hand-editing if needed, and then using the KFG OODB(Object-oriented Class Database) to Excel Writer to generate final Excel files. These Excel files can then be read, by a KFG Excel File Reader and then appended to the KFG OODB.

Code 
import glob
from ascher_excel_book import Ascher_Excel_Book
exceldir = Ascher_Excel_Book.ascher_excel_file_directory() 
ascher_excel_files = sorted(glob.glob(f"{exceldir}/AS*.xlsx"))
if len(ascher_excel_files) != 9: print("Could not import 9 Ascher Excel Files")
ascher_excel_books = [Ascher_Excel_Book(filename) for filename in ascher_excel_files ]

# A review of the sample proprietary dataframes generated
sample_book = ascher_excel_books[7]
sample_book.ascher_cord_cluster_df.head()
sample_book.ascher_cord_df.head()
sample_book.ascher_color_df.head()
sample_book.knot_cluster_df.head()
sample_book.knot_df.head()
cord_cluster_index khipu_id khipu_name cord_id cluster_id ordinal start_position end_position num_cords cluster_level ... beg_cord end_cord created_by created_on changed_by changed_on grouping_class beg_inv_cord end_inv_cord cord_range
0 4080001 4000008 AS008 4000008 4080001 0 85.0 85.5 2 1 ... 1 2 marciaascher 12/31/72 ashokkhosla 09/01/21 0 0 1-2
1 4080002 4000008 AS008 4000008 4080002 1 105.5 105.5 1 1 ... 3 3 marciaascher 12/31/72 ashokkhosla 09/01/21 0 0 3
2 4080003 4000008 AS008 4000008 4080003 2 109.0 118.0 1 1 ... 4 4 marciaascher 12/31/72 ashokkhosla 09/01/21 0 0 4
3 4080004 4000008 AS008 4000008 4080004 3 121.0 121.5 3 1 ... 5 7 marciaascher 12/31/72 ashokkhosla 09/01/21 0 0 5-7
4 4080005 4000008 AS008 4000008 4080005 4 122.5 123.0 3 1 ... 8 10 marciaascher 12/31/72 ashokkhosla 09/01/21 0 0 8-10

5 rows Ă— 21 columns

cord_index khipu_id khipu_name cord_id twist_angle thickness fiber termination pendant_from attached_to ... cord_notes pendant_index twist cord_classification knots full_color cord_value investigator_cord_num canuto_ordinal canuto_id
0 4080001 4000008 AS008 4080001 0 0 4000008 0 ... 1 U 6L(22.0) LB 6.0 1 0 0
1 4080007 4000008 AS008 4080007 0 0 4000008 0 ... 2 U 3L(20.0) W:LB 3.0 7 0 0
2 4080008 4000008 AS008 4080008 0 0 4000008 0 ... 3 U 6S(8.0);6S(13.0);8L(18.0) LB 668.0 8 0 0
3 4080010 4000008 AS008 4080010 0 0 4000008 0 ... 4 U 3S(10.0);3S(16.0);1E(22.0) W 331.0 10 0 0
4 4080011 4000008 AS008 4080011 0 0 4000008 0 ... 5 U MB 0.0 11 0 0

5 rows Ă— 29 columns

color_index color_id khipu_id cord_id pcord_flag color_cd_1 operator_1 color_range pigmentation_cd_1 color_cd_2 ... range_end pigmentation_cd_2 pigmentation_cd_3 pigmentation_cd_4 pigmentation_cd_5 full_color changed_by changed_on created_by created_on
0 4080001 4080001 4000008 4080001 0 LB ... 49.0 LB ashokkhosla 09/01/21 marciaascher 12/31/72
1 4080002 4080002 4000008 4080007 0 W ... 41.0 W:LB ashokkhosla 09/01/21 marciaascher 12/31/72
2 4080003 4080003 4000008 4080008 0 LB ... 33.0 LB ashokkhosla 09/01/21 marciaascher 12/31/72
3 4080004 4080004 4000008 4080010 0 W ... 40.0 W ashokkhosla 09/01/21 marciaascher 12/31/72
4 4080005 4080005 4000008 4080011 0 MB ... 37.0 MB ashokkhosla 09/01/21 marciaascher 12/31/72

5 rows Ă— 28 columns

knot_cluster_index cord_id cluster_id start_pos end_pos total_value ordinal num_knots notes created_by created_on changed_by changed_on
0 4080001 4080001 4080001 22.0 0 6 1 6 marciaascher 12/31/72 ashokkhosla 09/01/21
1 4080002 4080007 4080002 20.0 0 3 1 3 marciaascher 12/31/72 ashokkhosla 09/01/21
2 4080003 4080008 4080003 8.0 0 600 1 6 marciaascher 12/31/72 ashokkhosla 09/01/21
3 4080004 4080008 4080004 13.0 0 60 2 6 marciaascher 12/31/72 ashokkhosla 09/01/21
4 4080005 4080008 4080005 18.0 0 8 3 8 marciaascher 12/31/72 ashokkhosla 09/01/21
knot_index cord_id type_code direction knot_value_type knot_value_decimal cluster_id knot_id knot_ordinal cluster_ordinal created_by created_on changed_by changed_on num_turns axis_orientation
0 4080001 4080001 L 6 6 4080001 4080001 1 1 marciaascher 12/31/72 ashokkhosla 09/01/21 6
1 4080002 4080007 L 3 3 4080002 4080002 1 1 marciaascher 12/31/72 ashokkhosla 09/01/21 3
2 4080003 4080008 S 100 100 4080003 4080003 1 1 marciaascher 12/31/72 ashokkhosla 09/01/21 6
3 4080004 4080008 S 100 100 4080003 4080004 2 1 marciaascher 12/31/72 ashokkhosla 09/01/21 6
4 4080005 4080008 S 100 100 4080003 4080005 3 1 marciaascher 12/31/72 ashokkhosla 09/01/21 6

The final step:

The final step was performed in the Database Review/Build Page: >Ascher_Excel_Book.append_ascher_dataframes()
print(“Added Ascher AS001-AS009 - Rebuilding khipu object-oriented database”)

However, since the addition of the KFG_Excel_Writer class, this has changed. With the ability to output database khipu to an Excel spreadsheet and then re-import that spreadsheet without any data loss, the preferred method is to import the Ascher khipus from a standard KFG Excel template file.

Code 
# from ascher_excel_book import Ascher_Excel_Book
# Ascher_Excel_Book.append_ascher_dataframes()
# print("Added Ascher AS001-AS009 - Rebuilding khipu object-oriented database")

# kq.refresh_database()
# kamayuq.rebuild_khipu_OODB()
# kq.build_qollqa()

# all_khipus = [aKhipu for aKhipu in kamayuq.fetch_all_khipus(clean_build=False,).values()]
# print(f"Done - processed {len(all_khipus)} khipus")