Build #39154

Environment variables

NameValue
ANDROID_HOME/home/android-sdk/
AWS_ACCESS_KEY_ID[*******]
AWS_SECRET_ACCESS_KEY[*******]
BUILD_CAUSEGHPRBCAUSE
BUILD_CAUSE_GHPRBCAUSEtrue
BUILD_DISPLAY_NAME#39154
BUILD_ID39154
BUILD_NUMBER39154
BUILD_TAGjenkins-SparkPullRequestBuilder-K8s-39154
BUILD_URLhttps://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39154/
CLASSPATH$CLASSPATH
DBUS_SESSION_BUS_ADDRESSunix:path=/run/user/1001/bus
EXECUTOR_NUMBER3
GITHUB_OAUTH_KEY[*******]
GIT_BRANCHupdated_28618
GIT_COMMITb2e57711ec13fc057ed10ab09b6ae888435a3297
GIT_PREVIOUS_COMMIT576f3411d6888c8a6202e05fdaa463f0cd5a895d
GIT_PREVIOUS_SUCCESSFUL_COMMIT576f3411d6888c8a6202e05fdaa463f0cd5a895d
GIT_URLhttps://github.com/apache/spark.git
HOME/home/jenkins
HUDSON_HOME/var/lib/jenkins
HUDSON_SERVER_COOKIE472906e9832aeb79
HUDSON_URLhttps://amplab.cs.berkeley.edu/jenkins/
JAVA_HOME/usr/java/latest
JENKINS_HOME/var/lib/jenkins
JENKINS_SERVER_COOKIE472906e9832aeb79
JENKINS_URLhttps://amplab.cs.berkeley.edu/jenkins/
JOB_BASE_NAMESparkPullRequestBuilder-K8s
JOB_NAMESparkPullRequestBuilder-K8s
JOB_URLhttps://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/
LANGen_US.UTF-8
LOGNAMEjenkins
MOTD_SHOWNpam
NODE_LABELSresearch-jenkins-worker-01 ubuntu ubuntu20 worker-01
NODE_NAMEresearch-jenkins-worker-01
OLDPWD/home/jenkins
PATH/home/jenkins/tools/hudson.tasks.Maven_MavenInstallation/Maven_3.6.3/bin/:/home/jenkins/gems/bin:/usr/local/go/bin:/home/jenkins/go-projects/bin:/home/jenkins/anaconda2/bin:/home/jenkins/tools/hudson.tasks.Maven_MavenInstallation/Maven_3.6.3/bin/:/home/jenkins/gems/bin:/usr/local/go/bin:/home/jenkins/go-projects/bin:/home/jenkins/anaconda2/bin:$PATH
PWD/home/jenkins
ROOT_BUILD_CAUSEGHPRBCAUSE
ROOT_BUILD_CAUSE_GHPRBCAUSEtrue
RUN_ARTIFACTS_DISPLAY_URLhttps://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39154/display/redirect?page=artifacts
RUN_CHANGES_DISPLAY_URLhttps://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39154/display/redirect?page=changes
RUN_DISPLAY_URLhttps://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39154/display/redirect
RUN_TESTS_DISPLAY_URLhttps://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39154/display/redirect?page=tests
SHELL/bin/bash
SHLVL0
SSH_CLIENT192.168.10.11 55878 22
SSH_CONNECTION192.168.10.11 55878 192.168.10.21 22
USERjenkins
WORKSPACE/home/jenkins/workspace/SparkPullRequestBuilder-K8s
XDG_RUNTIME_DIR/run/user/1001
XDG_SESSION_CLASSuser
XDG_SESSION_ID3892
XDG_SESSION_TYPEtty
_/usr/java/latest/bin/java
ghprbActualCommitb350258c88dfd3eddb9392be5b85a3390a2d39e5
ghprbActualCommitAuthor“attilapiros”
ghprbActualCommitAuthorEmailpiros.attila.zsolt@gmail.com
ghprbAuthorRepoGitUrlhttps://github.com/attilapiros/spark.git
ghprbCommentBodynull
ghprbCredentialsIdb7d94526-9e9b-435f-9275-d7dbf209f4a3
ghprbGhRepositoryapache/spark
ghprbPullAuthorEmail
ghprbPullAuthorLoginattilapiros
ghprbPullAuthorLoginMention@attilapiros
ghprbPullDescriptionGitHub pull request #30763 of commit b350258c88dfd3eddb9392be5b85a3390a2d39e5, no merge conflicts.
ghprbPullId30763
ghprbPullLinkhttps://github.com/apache/spark/pull/30763
ghprbPullLongDescriptionThis is a copy of #28618 but merged with the current master resolving all the merge conflicts. \r\nAll the credit goes to @mccheah I just would like to help out here and avoid his progress to be lost. \r\n\r\n### What changes were proposed in this pull request?\r\n\r\nAdds a `ShuffleOutputTracker` API that can be used for managing shuffle metadata on the driver. Accepts map output metadata returned by the map output writers.\r\n\r\nRequires #28616.\r\n\r\n### Why are the changes needed?\r\nPart of the design as discussed in this document, and part of the wider effort of SPARK-25299.\r\n\r\n### Does this PR introduce any user-facing change?\r\n\r\nEnables additional APIs for the shuffle storage plugin tree. Usage will become more apparent when the read side of the shuffle plugin tree is introduced.\r\n\r\n### How was this patch tested?\r\n\r\nWe've added a mock implementation of the shuffle plugin tree here, to prove that a Spark job using a different implementation of the plugin can use all of the plugin points for an alternative shuffle data storage solution. But we don't include it here, in order to minimize the diff and the code to review in this specific patch. See #28902.
ghprbPullTitle[SPARK-31801][API][SHUFFLE] Register map output metadata
ghprbSourceBranchupdated_28618
ghprbTargetBranchmaster
ghprbTriggerAuthor
ghprbTriggerAuthorEmail
ghprbTriggerAuthorLogin
ghprbTriggerAuthorLoginMention
sha1origin/pr/30763/merge